Fix CI

Allow strict document update without creating missing documents

.github/ISSUE_TEMPLATE/new_feature_issue.md

@@ -24,6 +24,11 @@ TBD
 - [ ] If not, add the `no db change` label to your PR, and you're good to merge.
 - [ ] If yes, add the `db change` label to your PR. You'll receive a message explaining you what to do.
 
+### Reminders when adding features
+
+- [ ] Write unit tests using insta
+- [ ] Write declarative integration tests in [workloads/tests](https://github.com/meilisearch/meilisearch/tree/main/workloads/test). Specify the routes to call and then call `cargo xtask test workloads/tests/YOUR_TEST.json --update-responses` so that responses are automatically filled.
+
 ### Reminders when modifying the API
 
 - [ ] Update the openAPI file with utoipa:

.github/workflows/bench-manual.yml

@@ -18,7 +18,7 @@ jobs:
     timeout-minutes: 180 # 3h
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
         with:
           profile: minimal
 

.github/workflows/bench-pr.yml

@@ -66,9 +66,7 @@ jobs:
           fetch-depth: 0 # fetch full history to be able to get main commit sha
           ref: ${{ steps.comment-branch.outputs.head_ref }}
 
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
-        with:
-          profile: minimal
 
       - name: Run benchmarks on PR ${{ github.event.issue.id }}
         run: |

.github/workflows/bench-push-indexing.yml

@@ -12,9 +12,7 @@ jobs:
     timeout-minutes: 180 # 3h
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
-        with:
-          profile: minimal
 
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch main - Commit ${{ github.sha }}

.github/workflows/benchmarks-manual.yml

@@ -18,7 +18,7 @@ jobs:
     timeout-minutes: 4320 # 72h
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
         with:
           profile: minimal
 

.github/workflows/benchmarks-pr.yml

@@ -44,7 +44,7 @@ jobs:
             exit 1
           fi
 
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
         with:
           profile: minimal
 

.github/workflows/benchmarks-push-indexing.yml

@@ -16,7 +16,7 @@ jobs:
     timeout-minutes: 4320 # 72h
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
         with:
           profile: minimal
 

.github/workflows/benchmarks-push-search-geo.yml

@@ -15,7 +15,7 @@ jobs:
     runs-on: benchmarks
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
         with:
           profile: minimal
 

.github/workflows/benchmarks-push-search-songs.yml

@@ -15,7 +15,7 @@ jobs:
     runs-on: benchmarks
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
         with:
           profile: minimal
 

.github/workflows/benchmarks-push-search-wiki.yml

@@ -15,7 +15,7 @@ jobs:
     runs-on: benchmarks
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
         with:
           profile: minimal
 

.github/workflows/db-change-comments.yml

@@ -6,7 +6,7 @@ on:
 
 env:
   MESSAGE: |
-    ### Hello, I'm a bot 🤖 
+    ### Hello, I'm a bot 🤖
 
     You are receiving this message because you declared that this PR make changes to the Meilisearch database.
     Depending on the nature of the change, additional actions might be required on your part. The following sections detail the additional actions depending on the nature of the change, please copy the relevant section in the description of your PR, and make sure to perform the required actions.
@@ -19,6 +19,7 @@ env:
 
     - [ ] Detail the change to the DB format and why they are forward compatible
     - [ ] Forward-compatibility: A database created before this PR and using the features touched by this PR was able to be opened by a Meilisearch produced by the code of this PR.
+    - [ ] Declarative test: add a [declarative test containing a dumpless upgrade](https://github.com/meilisearch/meilisearch/blob/main/TESTING.md#typical-usage)
 
 
     ## This PR makes breaking changes
@@ -35,8 +36,7 @@ env:
       - [ ] Write the code to go from the old database to the new one
         - If the change happened in milli, the upgrade function should be written and called [here](https://github.com/meilisearch/meilisearch/blob/3fd86e8d76d7d468b0095d679adb09211ca3b6c0/crates/milli/src/update/upgrade/mod.rs#L24-L47)
         - If the change happened in the index-scheduler, we've never done it yet, but the right place to do it should be [here](https://github.com/meilisearch/meilisearch/blob/3fd86e8d76d7d468b0095d679adb09211ca3b6c0/crates/index-scheduler/src/scheduler/process_upgrade/mod.rs#L13)
-      - [ ] Write an integration test [here](https://github.com/meilisearch/meilisearch/blob/main/crates/meilisearch/tests/upgrade/mod.rs) ensuring you can read the old database, upgrade to the new database, and read the new database as expected
+      - [ ] Declarative test: add a [declarative test containing a dumpless upgrade](https://github.com/meilisearch/meilisearch/blob/main/TESTING.md#typical-usage)
-    
 
 jobs:
   add-comment:

.github/workflows/flaky-tests.yml

@@ -3,7 +3,7 @@ name: Look for flaky tests
 on:
   workflow_dispatch:
   schedule:
-    - cron: '0 4 * * *' # Every day at 4:00AM
+    - cron: "0 4 * * *" # Every day at 4:00AM
 
 jobs:
   flaky:
@@ -13,11 +13,17 @@ jobs:
       image: ubuntu:22.04
     steps:
       - uses: actions/checkout@v5
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
+        run: |
+          sudo rm -rf "/opt/ghc" || true
+          sudo rm -rf "/usr/share/dotnet" || true
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
       - name: Install needed dependencies
         run: |
           apt-get update && apt-get install -y curl
           apt-get install build-essential -y
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
       - name: Install cargo-flaky
         run: cargo install cargo-flaky
       - name: Run cargo flaky in the dumps

.github/workflows/fuzzer-indexing.yml

@@ -12,9 +12,7 @@ jobs:
     timeout-minutes: 4320 # 72h
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
-        with:
-          profile: minimal
 
       # Run benchmarks
       - name: Run the fuzzer

.github/workflows/publish-apt-brew-pkg.yml

@@ -25,7 +25,13 @@ jobs:
         run: |
           apt-get update && apt-get install -y curl
           apt-get install build-essential -y
-      - uses: dtolnay/rust-toolchain@1.89
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
+        run: |
+          sudo rm -rf "/opt/ghc" || true
+          sudo rm -rf "/usr/share/dotnet" || true
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
+      - uses: dtolnay/rust-toolchain@1.91.1
       - name: Install cargo-deb
         run: cargo install cargo-deb
       - uses: actions/checkout@v5

.github/workflows/publish-docker-images.yml

@@ -208,8 +208,8 @@ jobs:
           done
           cosign sign --yes ${images}
 
-      # /!\ Don't touch this without checking with Cloud team
+      # /!\ Don't touch this without checking with engineers working on the Cloud code base on #discussion-engineering Slack channel
-      - name: Send CI information to Cloud team
+      - name: Notify meilisearch-cloud 
         # Do not send if nightly build (i.e. 'schedule' or 'workflow_dispatch' event)
         if: ${{ (github.event_name == 'push') && (matrix.edition == 'enterprise') }}
         uses: peter-evans/repository-dispatch@v3
@@ -218,3 +218,14 @@ jobs:
           repository: meilisearch/meilisearch-cloud
           event-type: cloud-docker-build
           client-payload: '{ "meilisearch_version": "${{ github.ref_name }}", "stable": "${{ steps.check-tag-format.outputs.stable }}" }'
+
+      # /!\ Don't touch this without checking with integration team members on #discussion-integrations Slack channel 
+      - name: Notify meilisearch-kubernetes
+        # Do not send if nightly build (i.e. 'schedule' or 'workflow_dispatch' event), or if not stable
+        if: ${{ github.event_name == 'push' && matrix.edition == 'community' && steps.check-tag-format.outputs.stable == 'true' }}
+        uses: peter-evans/repository-dispatch@v3
+        with:
+          token: ${{ secrets.MEILI_BOT_GH_PAT }}
+          repository: meilisearch/meilisearch-kubernetes
+          event-type: meilisearch-release
+          client-payload: '{ "version": "${{ github.ref_name }}" }'

.github/workflows/publish-release-assets.yml

@@ -76,7 +76,7 @@ jobs:
     needs: check-version
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
       - name: Build
         run: cargo build --release --locked ${{ matrix.feature-flag }} ${{ matrix.extra-args }}
       # No need to upload binaries for dry run (cron or workflow_dispatch)

.github/workflows/sdks-tests.yml

@@ -25,14 +25,18 @@ jobs:
       - uses: actions/checkout@v5
       - name: Define the Docker image we need to use
         id: define-image
+        env:
+          EVENT_NAME: ${{ github.event_name }}
+          DOCKER_IMAGE_INPUT: ${{ github.event.inputs.docker_image }}
         run: |
-          event=${{ github.event_name }}
           echo "docker-image=nightly" >> $GITHUB_OUTPUT
-          if [[ $event == 'workflow_dispatch' ]]; then
+          if [[ "$EVENT_NAME" == 'workflow_dispatch' ]]; then
-            echo "docker-image=${{ github.event.inputs.docker_image }}" >> $GITHUB_OUTPUT
+            echo "docker-image=$DOCKER_IMAGE_INPUT" >> $GITHUB_OUTPUT
           fi
       - name: Docker image is ${{ steps.define-image.outputs.docker-image }}
-        run: echo "Docker image is ${{ steps.define-image.outputs.docker-image }}"
+        env:
+          DOCKER_IMAGE: ${{ steps.define-image.outputs.docker-image }}
+        run: echo "Docker image is $DOCKER_IMAGE"
 
 ##########
 ## SDKs ##
@@ -159,24 +163,26 @@ jobs:
       - uses: actions/checkout@v5
         with:
           repository: meilisearch/meilisearch-js
+      - name: Install pnpm
+        uses: pnpm/action-setup@v4
       - name: Setup node
-        uses: actions/setup-node@v5
+        uses: actions/setup-node@v6
         with:
-          cache: 'yarn'
+          cache: 'pnpm'
       - name: Install dependencies
-        run: yarn --dev
+        run: pnpm install
       - name: Run tests
-        run: yarn test
+        run: pnpm test
       - name: Build project
-        run: yarn build
+        run: pnpm build
       - name: Run ESM env
-        run: yarn test:env:esm
+        run: pnpm test:env:esm
       - name: Run Node.js env
-        run: yarn test:env:nodejs
+        run: pnpm test:env:nodejs
       - name: Run node typescript env
-        run: yarn test:env:node-ts
+        run: pnpm test:env:node-ts
       - name: Run Browser env
-        run: yarn test:env:browser
+        run: pnpm test:env:browser
 
   meilisearch-php-tests:
     needs: define-docker-image

.github/workflows/test-suite.yml

@@ -19,88 +19,120 @@ jobs:
     runs-on: ${{ matrix.runner }}
     strategy:
       matrix:
-        runner: [ubuntu-24.04, ubuntu-24.04-arm]
+        runner: [ubuntu-22.04, ubuntu-22.04-arm]
         features: ["", "--features enterprise"]
-    container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
     steps:
       - uses: actions/checkout@v5
-      - name: Install needed dependencies
+      - name: check free space before
+        run: df -h
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
         run: |
-          apt-get update && apt-get install -y curl
+          sudo rm -rf "/opt/ghc" || true
-          apt-get install build-essential -y
+          sudo rm -rf "/usr/share/dotnet" || true
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
+      - name: check free space after
+        run: df -h
       - name: Setup test with Rust stable
-        uses: dtolnay/rust-toolchain@1.89
+        uses: dtolnay/rust-toolchain@1.91.1
       - name: Cache dependencies
         uses: Swatinem/rust-cache@v2.8.0
-      - name: Run cargo check without any default features
+        with:
+          key: ${{ matrix.features }}
+      - name: Run cargo build without any default features
         uses: actions-rs/cargo@v1
         with:
           command: build
-          args: --locked --release --no-default-features --all
+          args: --locked --no-default-features --all
       - name: Run cargo test
         uses: actions-rs/cargo@v1
         with:
           command: test
-          args: --locked --release --all ${{ matrix.features }}
+          args: --locked --all ${{ matrix.features }}
 
-  test-others:
+  test-windows:
     name: Tests on ${{ matrix.os }}
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
-        os: [macos-14, windows-2022]
+        os: [windows-2022]
         features: ["", "--features enterprise"]
-    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request'
+    if: github.event_name != 'merge_group'
     steps:
       - uses: actions/checkout@v5
       - name: Cache dependencies
         uses: Swatinem/rust-cache@v2.8.0
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: dtolnay/rust-toolchain@1.91.1
-      - name: Run cargo check without any default features
+      - name: Run cargo build without any default features
         uses: actions-rs/cargo@v1
         with:
           command: build
-          args: --locked --release --no-default-features --all
+          args: --locked --no-default-features --all
       - name: Run cargo test
         uses: actions-rs/cargo@v1
         with:
           command: test
-          args: --locked --release --all ${{ matrix.features }}
+          args: --locked --all ${{ matrix.features }}
 
-  test-all-features:
+  test-macos:
-    name: Tests almost all features
+    name: Tests on ${{ matrix.os }}
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
-    container:
+    strategy:
-      # Use ubuntu-22.04 to compile with glibc 2.35
+      fail-fast: false
-      image: ubuntu:22.04
+      matrix:
+        os: [macos-14]
+        features: ["", "--features enterprise"]
     if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
     steps:
       - uses: actions/checkout@v5
-      - name: Install needed dependencies
+      - name: Cache dependencies
+        uses: Swatinem/rust-cache@v2.8.0
+      - uses: dtolnay/rust-toolchain@1.91.1
+      - name: Run cargo build without any default features
+        uses: actions-rs/cargo@v1
+        with:
+          command: build
+          args: --locked --no-default-features --all
+      - name: Run cargo test
+        uses: actions-rs/cargo@v1
+        with:
+          command: test
+          args: --locked --all ${{ matrix.features }}
+
+  test-all-features:
+    name: Tests almost all features
+    runs-on: ubuntu-22.04
+    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+    steps:
+      - uses: actions/checkout@v5
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
         run: |
-          apt-get update
+          sudo rm -rf "/opt/ghc" || true
-          apt-get install --assume-yes build-essential curl
+          sudo rm -rf "/usr/share/dotnet" || true
-      - uses: dtolnay/rust-toolchain@1.89
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
+      - uses: dtolnay/rust-toolchain@1.91.1
       - name: Run cargo build with almost all features
         run: |
-          cargo build --workspace --locked --release --features "$(cargo xtask list-features --exclude-feature cuda,test-ollama)"
+          cargo build --workspace --locked --features "$(cargo xtask list-features --exclude-feature cuda,test-ollama)"
       - name: Run cargo test with almost all features
         run: |
-          cargo test --workspace --locked --release --features "$(cargo xtask list-features --exclude-feature cuda,test-ollama)"
+          cargo test --workspace --locked --features "$(cargo xtask list-features --exclude-feature cuda,test-ollama)"
 
   ollama-ubuntu:
     name: Test with Ollama
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
-    strategy:
-      matrix:
-        features: ["", "--features enterprise"]
     env:
       MEILI_TEST_OLLAMA_SERVER: "http://localhost:11434"
     steps:
       - uses: actions/checkout@v5
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
+        run: |
+          sudo rm -rf "/opt/ghc" || true
+          sudo rm -rf "/usr/share/dotnet" || true
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
       - name: Install Ollama
         run: |
           curl -fsSL https://ollama.com/install.sh | sudo -E sh
@@ -124,21 +156,21 @@ jobs:
         uses: actions-rs/cargo@v1
         with:
           command: test
-          args: --locked --release --all --features test-ollama ollama ${{ matrix.features }}
+          args: --locked -p meilisearch --features test-ollama ollama
 
   test-disabled-tokenization:
     name: Test disabled tokenization
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
-    container:
-      image: ubuntu:22.04
     if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
     steps:
       - uses: actions/checkout@v5
-      - name: Install needed dependencies
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
         run: |
-          apt-get update
+          sudo rm -rf "/opt/ghc" || true
-          apt-get install --assume-yes build-essential curl
+          sudo rm -rf "/usr/share/dotnet" || true
-      - uses: dtolnay/rust-toolchain@1.89
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
+      - uses: dtolnay/rust-toolchain@1.91.1
       - name: Run cargo tree without default features and check lindera is not present
         run: |
           if cargo tree -f '{p} {f}' -e normal --no-default-features | grep -qz lindera; then
@@ -149,33 +181,39 @@ jobs:
         run: |
           cargo tree -f '{p} {f}' -e normal | grep lindera -qz
 
-  # We run tests in debug also, to make sure that the debug_assertions are hit
+  build:
-  test-debug:
+    name: Build in release
-    name: Run tests in debug
+    runs-on: ubuntu-22.04
+    steps:
+      - uses: actions/checkout@v5
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
+        run: |
+          sudo rm -rf "/opt/ghc" || true
+          sudo rm -rf "/usr/share/dotnet" || true
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
+      - uses: dtolnay/rust-toolchain@1.91.1
+      - name: Cache dependencies
+        uses: Swatinem/rust-cache@v2.8.0
+      - name: Build
+        run: cargo build --release --locked --target x86_64-unknown-linux-gnu
+
+  clippy:
+    name: Run Clippy
     runs-on: ubuntu-22.04
     strategy:
       matrix:
         features: ["", "--features enterprise"]
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
-      - name: Run tests in debug
+        run: |
-        uses: actions-rs/cargo@v1
+          sudo rm -rf "/opt/ghc" || true
+          sudo rm -rf "/usr/share/dotnet" || true
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
+      - uses: dtolnay/rust-toolchain@1.91.1
         with:
-          command: test
-          args: --locked --all ${{ matrix.features }}
-
-  clippy:
-    name: Run Clippy
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        features: ["", "--features enterprise"]
-    steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
-        with:
-          profile: minimal
           components: clippy
       - name: Cache dependencies
         uses: Swatinem/rust-cache@v2.8.0
@@ -183,18 +221,21 @@ jobs:
         uses: actions-rs/cargo@v1
         with:
           command: clippy
-          args: --all-targets ${{ matrix.features }} -- --deny warnings
+          args: --all-targets ${{ matrix.features }} -- --deny warnings -D clippy::todo
 
   fmt:
     name: Run Rustfmt
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
+        run: |
+          sudo rm -rf "/opt/ghc" || true
+          sudo rm -rf "/usr/share/dotnet" || true
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
+      - uses: dtolnay/rust-toolchain@1.91.1
         with:
-          profile: minimal
-          toolchain: nightly-2024-07-09
-          override: true
           components: rustfmt
       - name: Cache dependencies
         uses: Swatinem/rust-cache@v2.8.0
@@ -205,3 +246,23 @@ jobs:
         run: |
           echo -ne "\n" > crates/benchmarks/benches/datasets_paths.rs
           cargo fmt --all -- --check
+
+  declarative-tests:
+    name: Run declarative tests
+    runs-on: ubuntu-22.04-arm
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v5
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
+        run: |
+          sudo rm -rf "/opt/ghc" || true
+          sudo rm -rf "/usr/share/dotnet" || true
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
+      - uses: dtolnay/rust-toolchain@1.91.1
+      - name: Cache dependencies
+        uses: Swatinem/rust-cache@v2.8.0
+      - name: Run declarative tests
+        run: |
+          cargo xtask test workloads/tests/*.json

.github/workflows/update-cargo-toml-version.yml

@@ -18,9 +18,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - name: Clean space as per https://github.com/actions/virtual-environments/issues/709
-        with:
+        run: |
-          profile: minimal
+          sudo rm -rf "/opt/ghc" || true
+          sudo rm -rf "/usr/share/dotnet" || true
+          sudo rm -rf "/usr/local/lib/android" || true
+          sudo rm -rf "/usr/local/share/boost" || true
+      - uses: dtolnay/rust-toolchain@1.91.1
       - name: Install sd
         run: cargo install sd
       - name: Update Cargo.toml file

BENCHMARKS.md

@@ -124,6 +124,7 @@ They are JSON files with the following structure (comments are not actually supp
 {
   // Name of the workload. Must be unique to the workload, as it will be used to group results on the dashboard.
   "name": "hackernews.ndjson_1M,no-threads",
+  "type": "bench",
   // Number of consecutive runs of the commands that should be performed.
   // Each run uses a fresh instance of Meilisearch and a fresh database.
   // Each run produces its own report file.

Cargo.lock

@@ -580,7 +580,7 @@ source = "git+https://github.com/meilisearch/bbqueue#e8af4a4bccc8eb36b2b0442c4a9
 
 [[package]]
 name = "benchmarks"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "anyhow",
  "bumpalo",
@@ -790,11 +790,11 @@ dependencies = [
 
 [[package]]
 name = "build-info"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "anyhow",
  "time",
- "vergen-git2",
+ "vergen-gitcl",
 ]
 
 [[package]]
@@ -1786,7 +1786,7 @@ dependencies = [
 
 [[package]]
 name = "dump"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "anyhow",
  "big_s",
@@ -2018,7 +2018,7 @@ checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be"
 
 [[package]]
 name = "file-store"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "tempfile",
  "thiserror 2.0.17",
@@ -2040,7 +2040,7 @@ dependencies = [
 
 [[package]]
 name = "filter-parser"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "insta",
  "levenshtein_automata",
@@ -2068,7 +2068,7 @@ dependencies = [
 
 [[package]]
 name = "flatten-serde-json"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "criterion",
  "serde_json",
@@ -2231,7 +2231,7 @@ dependencies = [
 
 [[package]]
 name = "fuzzers"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "arbitrary",
  "bumpalo",
@@ -2604,19 +2604,6 @@ version = "0.32.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e629b9b98ef3dd8afe6ca2bd0f89306cec16d43d907889945bc5d6687f2f13c7"
 
-[[package]]
-name = "git2"
-version = "0.20.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2deb07a133b1520dc1a5690e9bd08950108873d7ed5de38dcc74d3b5ebffa110"
-dependencies = [
- "bitflags 2.10.0",
- "libc",
- "libgit2-sys",
- "log",
- "url",
-]
-
 [[package]]
 name = "glob"
 version = "0.3.3"
@@ -2711,9 +2698,9 @@ dependencies = [
 
 [[package]]
 name = "hannoy"
-version = "0.0.9-nested-rtxns-2"
+version = "0.1.0-nested-rtxns"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "06eda090938d9dcd568c8c2a5de383047ed9191578ebf4a342d2975d16e621f2"
+checksum = "be82bf3f2108ddc8885e3d306fcd7f4692066bfe26065ca8b42ba417f3c26dd1"
 dependencies = [
  "bytemuck",
  "byteorder",
@@ -3198,7 +3185,7 @@ dependencies = [
 
 [[package]]
 name = "index-scheduler"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "anyhow",
  "backoff",
@@ -3216,6 +3203,7 @@ dependencies = [
  "enum-iterator",
  "file-store",
  "flate2",
+ "hashbrown 0.15.5",
  "indexmap",
  "insta",
  "maplit",
@@ -3238,6 +3226,7 @@ dependencies = [
  "tokio",
  "tracing",
  "ureq",
+ "urlencoding",
  "uuid",
 ]
 
@@ -3460,7 +3449,7 @@ dependencies = [
 
 [[package]]
 name = "json-depth-checker"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "criterion",
  "serde_json",
@@ -3557,18 +3546,6 @@ dependencies = [
  "rle-decode-fast",
 ]
 
-[[package]]
-name = "libgit2-sys"
-version = "0.18.2+1.9.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1c42fe03df2bd3c53a3a9c7317ad91d80c81cd1fb0caec8d7cc4cd2bfa10c222"
-dependencies = [
- "cc",
- "libc",
- "libz-sys",
- "pkg-config",
-]
-
 [[package]]
 name = "libloading"
 version = "0.8.9"
@@ -3626,18 +3603,6 @@ dependencies = [
  "zlib-rs",
 ]
 
-[[package]]
-name = "libz-sys"
-version = "1.1.22"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8b70e7a7df205e92a1a4cd9aaae7898dac0aa555503cc0a649494d0d60e7651d"
-dependencies = [
- "cc",
- "libc",
- "pkg-config",
- "vcpkg",
-]
-
 [[package]]
 name = "lindera"
 version = "0.43.3"
@@ -3974,7 +3939,7 @@ checksum = "ae960838283323069879657ca3de837e9f7bbb4c7bf6ea7f1b290d5e9476d2e0"
 
 [[package]]
 name = "meili-snap"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "insta",
  "md5 0.8.0",
@@ -3985,7 +3950,7 @@ dependencies = [
 
 [[package]]
 name = "meilisearch"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "actix-cors",
  "actix-http",
@@ -4083,7 +4048,7 @@ dependencies = [
 
 [[package]]
 name = "meilisearch-auth"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "base64 0.22.1",
  "enum-iterator",
@@ -4102,10 +4067,11 @@ dependencies = [
 
 [[package]]
 name = "meilisearch-types"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "actix-web",
  "anyhow",
+ "base64 0.22.1",
  "bumpalo",
  "bumparaw-collections",
  "byte-unit",
@@ -4118,6 +4084,7 @@ dependencies = [
  "flate2",
  "fst",
  "insta",
+ "itertools 0.14.0",
  "meili-snap",
  "memmap2",
  "milli",
@@ -4131,13 +4098,14 @@ dependencies = [
  "thiserror 2.0.17",
  "time",
  "tokio",
+ "urlencoding",
  "utoipa",
  "uuid",
 ]
 
 [[package]]
 name = "meilitool"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "anyhow",
  "clap",
@@ -4171,7 +4139,7 @@ dependencies = [
 
 [[package]]
 name = "milli"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "arroy",
  "bbqueue",
@@ -4750,7 +4718,7 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
 
 [[package]]
 name = "permissive-json-pointer"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "big_s",
  "serde_json",
@@ -6072,6 +6040,20 @@ name = "similar"
 version = "2.7.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "bbbb5d9659141646ae647b42fe094daf6c6192d1620870b449d9557f748b2daa"
+dependencies = [
+ "bstr",
+ "unicode-segmentation",
+]
+
+[[package]]
+name = "similar-asserts"
+version = "1.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b5b441962c817e33508847a22bd82f03a30cff43642dc2fae8b050566121eb9a"
+dependencies = [
+ "console",
+ "similar",
+]
 
 [[package]]
 name = "simple_asn1"
@@ -7105,12 +7087,6 @@ version = "0.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65"
 
-[[package]]
-name = "vcpkg"
-version = "0.2.15"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426"
-
 [[package]]
 name = "vergen"
 version = "9.0.6"
@@ -7124,14 +7100,13 @@ dependencies = [
 ]
 
 [[package]]
-name = "vergen-git2"
+name = "vergen-gitcl"
-version = "1.0.7"
+version = "1.0.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4f6ee511ec45098eabade8a0750e76eec671e7fb2d9360c563911336bea9cac1"
+checksum = "b9dfc1de6eb2e08a4ddf152f1b179529638bedc0ea95e6d667c014506377aefe"
 dependencies = [
  "anyhow",
  "derive_builder",
- "git2",
  "rustversion",
  "time",
  "vergen",
@@ -7783,7 +7758,7 @@ dependencies = [
 
 [[package]]
 name = "xtask"
-version = "1.28.2"
+version = "1.30.0"
 dependencies = [
  "anyhow",
  "build-info",
@@ -7792,9 +7767,11 @@ dependencies = [
  "futures-core",
  "futures-util",
  "reqwest",
+ "semver",
  "serde",
  "serde_json",
  "sha2",
+ "similar-asserts",
  "sysinfo",
  "time",
  "tokio",

Cargo.toml

@@ -23,7 +23,7 @@ members = [
 ]
 
 [workspace.package]
-version = "1.28.2"
+version = "1.30.0"
 authors = [
     "Quentin de Quelen <quentin@dequelen.me>",
     "Clément Renault <clement@meilisearch.com>",

README.md

@@ -104,6 +104,7 @@ Meilisearch is available in two editions:
 
 - Includes advanced features such as:
   - Sharding
+  - S3-streaming snapshots
 - Governed by a [commercial license](./LICENSE-EE) or the [Business Source License 1.1](https://mariadb.com/bsl11)
 - Not allowed in production without a commercial agreement with Meilisearch.
   - You may use, modify, and distribute the Licensed Work for non-production purposes only, such as testing, development, or evaluation.

TESTING.md

@@ -0,0 +1,326 @@
+# Declarative tests
+
+Declarative tests ensure that Meilisearch features remain stable across versions.
+
+While we already have unit tests, those are run against **temporary databases** that are created fresh each time and therefore never risk corruption.
+
+Declarative tests instead **simulate the lifetime of a database**: they chain together commands and requests to change the binary, verifying that database state and API responses remain consistent.
+
+## Basic example
+
+```jsonc
+{
+  "type": "test",
+  "name": "api-keys",
+  "binary": { // the first command will run on the binary following this specification.
+    "source": "release", // get the binary as a release from GitHub
+    "version": "1.19.0", // version to fetch
+    "edition": "community" // edition to fetch
+  },
+  "commands": []
+}
+```
+
+This example defines a no-op test (it does nothing).
+
+If the file is saved at `workloads/tests/example.json`, you can run it with:
+
+```bash
+cargo xtask test workloads/tests/example.json
+```
+
+## Commands
+
+Commands represent API requests sent to Meilisearch endpoints during a test.
+
+They are executed sequentially, and their responses can be validated to ensure consistent behavior across upgrades.
+
+```jsonc
+
+{
+  "route": "keys",
+  "method": "POST",
+  "body": {
+    "inline": {
+      "actions": [
+        "search",
+        "documents.add"
+      ],
+      "description": "Test API Key",
+      "expiresAt": null,
+      "indexes": [ "movies" ]
+    }
+  }
+}
+```
+
+This command issues a `POST /keys` request, creating an API key with permissions to search and add documents in the `movies` index.
+
+### Using assets in commands
+
+To keep tests concise and reusable, you can define **assets** at the root of the workload file.
+
+Assets are external data sources (such as datasets) that are cached between runs, making tests faster and easier to read.
+
+```jsonc
+{
+  "type": "test",
+  "name": "movies",
+  "binary": {
+    "source": "release",
+    "version": "1.19.0",
+    "edition": "community"
+  },
+  "assets": {
+    "movies.json": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/movies.json",
+      "sha256": "5b6e4cb660bc20327776e8a33ea197b43d9ec84856710ead1cc87ab24df77de1"
+    }
+  },
+  "commands": [
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "movies.json"
+      }
+    }
+  ]
+}
+```
+
+In this example:
+- The `movies.json` dataset is defined as an asset, pointing to a remote URL.
+- The SHA-256 checksum ensures integrity.
+- The `POST /indexes/movies/documents` command uses this asset as the request body.
+
+This makes the test much cleaner than inlining a large dataset directly into the command.
+
+For asset handling, please refer to the [declarative benchmarks documentation](/BENCHMARKS.md#adding-new-assets).
+
+### Asserting responses
+
+Commands can specify both the **expected status code** and the **expected response body**.
+
+```jsonc
+{
+  "route": "indexes/movies/documents",
+  "method": "POST",
+  "body": {
+    "asset": "movies.json"
+  },
+  "expectedStatus": 202,
+  "expectedResponse": {
+    "enqueuedAt": "[timestamp]", // Set to a bracketed string to ignore the value
+    "indexUid": "movies",
+    "status": "enqueued",
+    "taskUid": 1,
+    "type": "documentAdditionOrUpdate"
+  },
+  "synchronous": "WaitForTask"
+}
+```
+
+Manually writing `expectedResponse` fields can be tedious.
+
+Instead, you can let the test runner populate them automatically:
+
+```bash
+# Run the workload to populate expected fields. Only adds the missing ones, doesn't change existing data
+cargo xtask test workloads/tests/example.json --add-missing-responses
+
+# OR
+
+# Run the workload to populate expected fields. Updates all fields including existing ones
+cargo xtask test workloads/tests/example.json --update-responses
+```
+
+This workflow is recommended:
+
+1. Write the test without expected fields.
+2. Run it with `--add-missing-responses` to capture the actual responses.
+3. Review and commit the generated expectations.
+
+## Changing binary
+
+It is possible to insert an instruction to change the current Meilisearch instance from one binary specification to another during a test.
+
+When executed, such an instruction will:
+1. Stop the current Meilisearch instance.
+2. Fetch the binary specified by the instruction.
+3. Restart the server with the specified binary on the same database.
+
+```jsonc
+{
+  "type": "test",
+  "name": "movies",
+  "binary": {
+    "source": "release",
+    "version": "1.19.0", // start with version v1.19.0
+    "edition": "community"
+  },
+  "assets": {
+    "movies.json": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/movies.json",
+      "sha256": "5b6e4cb660bc20327776e8a33ea197b43d9ec84856710ead1cc87ab24df77de1"
+    }
+  },
+  "commands": [
+    // setup some data
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "movies.json"
+      }
+    },
+    // switch binary to v1.24.0
+    {
+      "binary": {
+        "source": "release",
+        "version": "1.24.0",
+        "edition": "community"
+      }
+    }
+  ]
+}
+```
+
+### Typical Usage
+
+In most cases, the change binary instruction will be used to update a database.
+
+- **Set up** some data using commands on an older version.
+- **Upgrade** to the latest version.
+- **Assert** that the data and API behavior remain correct after the upgrade.
+
+To properly test the dumpless upgrade, one should typically:
+
+1. Open the database without processing the update task: Use a `binary` instruction to switch to the desired version, passing `--experimental-dumpless-upgrade` and `--experimental-max-number-of-batched-tasks=0` as extra CLI arguments
+2. Check that the search, stats and task queue still work.
+3. Open the database and process the update task: Use a `binary` instruction to switch to the desired version, passing `--experimental-dumpless-upgrade` as the extra CLI argument. Use a `health` command to wait for the upgrade task to finish.
+4. Check that the indexing, search, stats, and task queue still work.
+
+```jsonc
+{
+  "type": "test",
+  "name": "movies",
+  "binary": {
+    "source": "release",
+    "version": "1.12.0",
+    "edition": "community"
+  },
+  "commands": [
+    // 0. Run commands to populate the database
+    {
+      // ..
+    },
+    // 1. Open the database with new MS without processing the update task
+    {
+      "binary": {
+        "source": "build", // build the binary from the sources in the current git repository
+        "edition": "community",
+        "extraCliArgs": [
+          "--experimental-dumpless-upgrade", // allows to open with a newer MS
+          "--experimental-max-number-of-batched-tasks=0" // prevent processing of the update task
+        ]
+      }
+    },
+    // 2. Check the search etc.
+    {
+      // ..
+    },
+    // 3. Open the database with new MS and processing the update task
+    {
+      "binary": {
+        "source": "build", // build the binary from the sources in the current git repository
+        "edition": "community",
+        "extraCliArgs": [
+          "--experimental-dumpless-upgrade" // allows to open with a newer MS
+          // no `--experimental-max-number-of-batched-tasks=0`
+        ]
+      }
+    },
+    // 4. Check the indexing, search, etc.
+    {
+      // ..
+    }
+  ]
+}
+```
+
+This ensures backward compatibility: databases created with older Meilisearch versions should remain functional and consistent after an upgrade.
+
+## Variables
+
+Sometimes a command needs to use a value returned by a **previous response**.
+These values can be captured and reused using the register field.
+
+```jsonc
+{
+  "route": "keys",
+  "method": "POST",
+  "body": {
+    "inline": {
+        "actions": [
+        "search",
+        "documents.add"
+        ],
+        "description": "Test API Key",
+        "expiresAt": null,
+        "indexes": [ "movies" ]
+    }
+  },
+  "expectedResponse": {
+      "key": "c6f64630bad2996b1f675007c8800168e14adf5d6a7bb1a400a6d2b158050eaf",
+      // ...
+  },
+  "register": {
+    "key": "/key"
+  },
+  "synchronous": "WaitForResponse"
+}
+```
+
+The `register` field captures the value at the JSON path `/key` from the response.
+Paths follow the **JavaScript Object Notation Pointer (RFC 6901)** format.
+Registered variables are available for all subsequent commands.
+
+Registered variables can be referenced by wrapping their name in double curly braces:
+
+In the route/path:
+
+```jsonc
+{
+  "route": "tasks/{{ task_id }}",
+  "method": "GET"
+}
+```
+
+In the request body:
+
+```jsonc
+{
+  "route": "indexes/movies/documents",
+  "method": "PATCH",
+  "body": {
+    "inline": {
+      "id": "{{ document_id }}",
+      "overview": "Shazam turns evil and the world is in danger.",
+    }
+  }
+}
+```
+
+Or they can be referenced by their name (**without curly braces**) as an API key:
+
+```jsonc
+{
+  "route": "indexes/movies/documents",
+  "method": "POST",
+  "body": { /* ... */ },
+  "apiKeyVariable": "key" // The **content** of the key variable will be used as an API key
+}
+```

crates/benchmarks/benches/indexing.rs

@@ -10,7 +10,7 @@ use milli::documents::PrimaryKey;
 use milli::heed::{EnvOpenOptions, RwTxn};
 use milli::progress::Progress;
 use milli::update::new::indexer;
-use milli::update::{IndexerConfig, Settings};
+use milli::update::{IndexerConfig, MissingDocumentPolicy, Settings};
 use milli::vector::RuntimeEmbedders;
 use milli::{FilterableAttributesRule, Index};
 use rand::seq::SliceRandom;
@@ -21,6 +21,10 @@ use roaring::RoaringBitmap;
 #[global_allocator]
 static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
 
+fn no_cancel() -> bool {
+    false
+}
+
 const BENCHMARK_ITERATION: usize = 10;
 
 fn setup_dir(path: impl AsRef<Path>) {
@@ -65,7 +69,7 @@ fn setup_settings<'t>(
     let sortable_fields = sortable_fields.iter().map(|s| s.to_string()).collect();
     builder.set_sortable_fields(sortable_fields);
 
-    builder.execute(&|| false, &Progress::default(), Default::default()).unwrap();
+    builder.execute(&no_cancel, &Progress::default(), Default::default()).unwrap();
 }
 
 fn setup_index_with_settings(
@@ -142,7 +146,7 @@ fn indexing_songs_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_SONGS, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -152,7 +156,7 @@ fn indexing_songs_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -168,7 +172,7 @@ fn indexing_songs_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -210,7 +214,7 @@ fn reindexing_songs_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_SONGS, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -220,7 +224,7 @@ fn reindexing_songs_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -236,7 +240,7 @@ fn reindexing_songs_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -256,7 +260,7 @@ fn reindexing_songs_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_SONGS, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -266,7 +270,7 @@ fn reindexing_songs_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -282,7 +286,7 @@ fn reindexing_songs_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -326,7 +330,7 @@ fn deleting_songs_in_batches_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_SONGS, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -336,7 +340,7 @@ fn deleting_songs_in_batches_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -352,7 +356,7 @@ fn deleting_songs_in_batches_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -404,7 +408,7 @@ fn indexing_songs_in_three_batches_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_SONGS_1_2, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -414,7 +418,7 @@ fn indexing_songs_in_three_batches_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -430,7 +434,7 @@ fn indexing_songs_in_three_batches_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -450,7 +454,7 @@ fn indexing_songs_in_three_batches_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_SONGS_3_4, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -460,7 +464,7 @@ fn indexing_songs_in_three_batches_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -476,7 +480,7 @@ fn indexing_songs_in_three_batches_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -492,7 +496,7 @@ fn indexing_songs_in_three_batches_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_SONGS_4_4, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -502,7 +506,7 @@ fn indexing_songs_in_three_batches_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -518,7 +522,7 @@ fn indexing_songs_in_three_batches_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -561,7 +565,7 @@ fn indexing_songs_without_faceted_numbers(c: &mut Criterion) {
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_SONGS, "csv");
 
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -571,7 +575,7 @@ fn indexing_songs_without_faceted_numbers(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -587,7 +591,7 @@ fn indexing_songs_without_faceted_numbers(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -629,7 +633,7 @@ fn indexing_songs_without_faceted_fields(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_SONGS, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -639,7 +643,7 @@ fn indexing_songs_without_faceted_fields(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -655,7 +659,7 @@ fn indexing_songs_without_faceted_fields(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -697,7 +701,7 @@ fn indexing_wiki(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_WIKI_ARTICLES, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -707,7 +711,7 @@ fn indexing_wiki(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -723,7 +727,7 @@ fn indexing_wiki(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -764,7 +768,7 @@ fn reindexing_wiki(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_WIKI_ARTICLES, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -774,7 +778,7 @@ fn reindexing_wiki(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -790,7 +794,7 @@ fn reindexing_wiki(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -810,7 +814,7 @@ fn reindexing_wiki(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_WIKI_ARTICLES, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -820,7 +824,7 @@ fn reindexing_wiki(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -836,7 +840,7 @@ fn reindexing_wiki(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -879,7 +883,7 @@ fn deleting_wiki_in_batches_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_WIKI_ARTICLES, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -889,7 +893,7 @@ fn deleting_wiki_in_batches_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -905,7 +909,7 @@ fn deleting_wiki_in_batches_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -957,7 +961,7 @@ fn indexing_wiki_in_three_batches(c: &mut Criterion) {
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents =
                     utils::documents_from(datasets_paths::SMOL_WIKI_ARTICLES_1_2, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -967,7 +971,7 @@ fn indexing_wiki_in_three_batches(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -983,7 +987,7 @@ fn indexing_wiki_in_three_batches(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1004,7 +1008,7 @@ fn indexing_wiki_in_three_batches(c: &mut Criterion) {
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents =
                     utils::documents_from(datasets_paths::SMOL_WIKI_ARTICLES_3_4, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1014,7 +1018,7 @@ fn indexing_wiki_in_three_batches(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1030,7 +1034,7 @@ fn indexing_wiki_in_three_batches(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1047,7 +1051,7 @@ fn indexing_wiki_in_three_batches(c: &mut Criterion) {
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents =
                     utils::documents_from(datasets_paths::SMOL_WIKI_ARTICLES_4_4, "csv");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1057,7 +1061,7 @@ fn indexing_wiki_in_three_batches(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1073,7 +1077,7 @@ fn indexing_wiki_in_three_batches(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1115,7 +1119,7 @@ fn indexing_movies_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::MOVIES, "json");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1125,7 +1129,7 @@ fn indexing_movies_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1141,7 +1145,7 @@ fn indexing_movies_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1182,7 +1186,7 @@ fn reindexing_movies_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::MOVIES, "json");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1192,7 +1196,7 @@ fn reindexing_movies_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1208,7 +1212,7 @@ fn reindexing_movies_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1228,7 +1232,7 @@ fn reindexing_movies_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::MOVIES, "json");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1238,7 +1242,7 @@ fn reindexing_movies_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1254,7 +1258,7 @@ fn reindexing_movies_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1297,7 +1301,7 @@ fn deleting_movies_in_batches_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::MOVIES, "json");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1307,7 +1311,7 @@ fn deleting_movies_in_batches_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1323,7 +1327,7 @@ fn deleting_movies_in_batches_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1372,7 +1376,7 @@ fn delete_documents_from_ids(index: Index, document_ids_to_delete: Vec<RoaringBi
             Some(primary_key),
             &document_changes,
             RuntimeEmbedders::default(),
-            &|| false,
+            &no_cancel,
             &Progress::default(),
             &Default::default(),
         )
@@ -1412,7 +1416,7 @@ fn indexing_movies_in_three_batches(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::MOVIES_1_2, "json");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1422,7 +1426,7 @@ fn indexing_movies_in_three_batches(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1438,7 +1442,7 @@ fn indexing_movies_in_three_batches(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1458,7 +1462,7 @@ fn indexing_movies_in_three_batches(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::MOVIES_3_4, "json");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1468,7 +1472,7 @@ fn indexing_movies_in_three_batches(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1484,7 +1488,7 @@ fn indexing_movies_in_three_batches(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1500,7 +1504,7 @@ fn indexing_movies_in_three_batches(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::MOVIES_4_4, "json");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1510,7 +1514,7 @@ fn indexing_movies_in_three_batches(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1526,7 +1530,7 @@ fn indexing_movies_in_three_batches(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1591,7 +1595,7 @@ fn indexing_nested_movies_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::NESTED_MOVIES, "json");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1601,7 +1605,7 @@ fn indexing_nested_movies_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1617,7 +1621,7 @@ fn indexing_nested_movies_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1683,7 +1687,7 @@ fn deleting_nested_movies_in_batches_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::NESTED_MOVIES, "json");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1693,7 +1697,7 @@ fn deleting_nested_movies_in_batches_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1709,7 +1713,7 @@ fn deleting_nested_movies_in_batches_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1767,7 +1771,7 @@ fn indexing_nested_movies_without_faceted_fields(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::NESTED_MOVIES, "json");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1777,7 +1781,7 @@ fn indexing_nested_movies_without_faceted_fields(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1793,7 +1797,7 @@ fn indexing_nested_movies_without_faceted_fields(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1835,7 +1839,7 @@ fn indexing_geo(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_ALL_COUNTRIES, "jsonl");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1845,7 +1849,7 @@ fn indexing_geo(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1861,7 +1865,7 @@ fn indexing_geo(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1902,7 +1906,7 @@ fn reindexing_geo(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_ALL_COUNTRIES, "jsonl");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1912,7 +1916,7 @@ fn reindexing_geo(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1928,7 +1932,7 @@ fn reindexing_geo(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -1948,7 +1952,7 @@ fn reindexing_geo(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_ALL_COUNTRIES, "jsonl");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -1958,7 +1962,7 @@ fn reindexing_geo(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -1974,7 +1978,7 @@ fn reindexing_geo(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )
@@ -2017,7 +2021,7 @@ fn deleting_geo_in_batches_default(c: &mut Criterion) {
 
                 let mut indexer = indexer::DocumentOperation::new();
                 let documents = utils::documents_from(datasets_paths::SMOL_ALL_COUNTRIES, "jsonl");
-                indexer.replace_documents(&documents).unwrap();
+                indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
                 let indexer_alloc = Bump::new();
                 let (document_changes, _operation_stats, primary_key) = indexer
@@ -2027,7 +2031,7 @@ fn deleting_geo_in_batches_default(c: &mut Criterion) {
                         &rtxn,
                         None,
                         &mut new_fields_ids_map,
-                        &|| false,
+                        &no_cancel,
                         Progress::default(),
                         None,
                     )
@@ -2043,7 +2047,7 @@ fn deleting_geo_in_batches_default(c: &mut Criterion) {
                     primary_key,
                     &document_changes,
                     RuntimeEmbedders::default(),
-                    &|| false,
+                    &no_cancel,
                     &Progress::default(),
                     &Default::default(),
                 )

crates/benchmarks/benches/utils.rs

@@ -13,7 +13,7 @@ use milli::documents::sort::recursive_sort;
 use milli::heed::EnvOpenOptions;
 use milli::progress::Progress;
 use milli::update::new::indexer;
-use milli::update::{IndexerConfig, Settings};
+use milli::update::{IndexerConfig, MissingDocumentPolicy, Settings};
 use milli::vector::RuntimeEmbedders;
 use milli::{Criterion, Filter, Index, Object, TermsMatchingStrategy};
 use serde_json::Value;
@@ -111,7 +111,7 @@ pub fn base_setup(conf: &Conf) -> Index {
 
     let documents = documents_from(conf.dataset, conf.dataset_format);
     let mut indexer = indexer::DocumentOperation::new();
-    indexer.replace_documents(&documents).unwrap();
+    indexer.replace_documents(&documents, MissingDocumentPolicy::default()).unwrap();
 
     let indexer_alloc = Bump::new();
     let (document_changes, _operation_stats, primary_key) = indexer

crates/build-info/Cargo.toml

@@ -15,4 +15,4 @@ time = { version = "0.3.44", features = ["parsing"] }
 
 [build-dependencies]
 anyhow = "1.0.100"
-vergen-git2 = "1.0.7"
+vergen-gitcl = "1.0.8"

crates/build-info/build.rs

@@ -15,7 +15,7 @@ fn emit_git_variables() -> anyhow::Result<()> {
     // Note: any code that needs VERGEN_ environment variables should take care to define them manually in the Dockerfile and pass them
     // in the corresponding GitHub workflow (publish_docker.yml).
     // This is due to the Dockerfile building the binary outside of the git directory.
-    let mut builder = vergen_git2::Git2Builder::default();
+    let mut builder = vergen_gitcl::GitclBuilder::default();
 
     builder.branch(true);
     builder.commit_timestamp(true);
@@ -25,5 +25,5 @@ fn emit_git_variables() -> anyhow::Result<()> {
 
     let git2 = builder.build()?;
 
-    vergen_git2::Emitter::default().fail_on_error().add_instructions(&git2)?.emit()
+    vergen_gitcl::Emitter::default().fail_on_error().add_instructions(&git2)?.emit()
 }

crates/build-info/src/main.rs

@@ -0,0 +1,6 @@
+use build_info::BuildInfo;
+
+fn main() {
+    let info = BuildInfo::from_build();
+    dbg!(info);
+}

crates/dump/src/lib.rs

@@ -9,8 +9,9 @@ use meilisearch_types::error::ResponseError;
 use meilisearch_types::keys::Key;
 use meilisearch_types::milli::update::IndexDocumentsMethod;
 use meilisearch_types::settings::Unchecked;
+use meilisearch_types::tasks::network::{DbTaskNetwork, NetworkTopologyChange};
 use meilisearch_types::tasks::{
-    Details, ExportIndexSettings, IndexSwap, KindWithContent, Status, Task, TaskId, TaskNetwork,
+    Details, ExportIndexSettings, IndexSwap, KindWithContent, Status, Task, TaskId,
 };
 use meilisearch_types::InstanceUid;
 use roaring::RoaringBitmap;
@@ -95,7 +96,7 @@ pub struct TaskDump {
     )]
     pub finished_at: Option<OffsetDateTime>,
     #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub network: Option<TaskNetwork>,
+    pub network: Option<DbTaskNetwork>,
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub custom_metadata: Option<String>,
 }
@@ -163,6 +164,7 @@ pub enum KindDump {
     IndexCompaction {
         index_uid: String,
     },
+    NetworkTopologyChange(NetworkTopologyChange),
 }
 
 impl From<Task> for TaskDump {
@@ -249,6 +251,9 @@ impl From<KindWithContent> for KindDump {
             KindWithContent::IndexCompaction { index_uid } => {
                 KindDump::IndexCompaction { index_uid }
             }
+            KindWithContent::NetworkTopologyChange(network_topology_change) => {
+                KindDump::NetworkTopologyChange(network_topology_change)
+            }
         }
     }
 }
@@ -560,7 +565,8 @@ pub(crate) mod test {
         Network {
             local: Some("myself".to_string()),
             remotes: maplit::btreemap! {"other".to_string() => Remote { url: "http://test".to_string(), search_api_key: Some("apiKey".to_string()), write_api_key: Some("docApiKey".to_string()) }},
-            sharding: false,
+            leader: None,
+            version: Default::default(),
         }
     }
 
@@ -614,7 +620,10 @@ pub(crate) mod test {
         assert_eq!(dump.features().unwrap().unwrap(), expected);
 
         // ==== checking the network
-        let expected = create_test_network();
+        let mut expected = create_test_network();
+        // from v1.29, we drop `leader` and `local` on import
+        expected.leader = None;
+        expected.local = None;
         assert_eq!(&expected, dump.network().unwrap().unwrap());
     }
 }

crates/dump/src/reader/mod.rs

@@ -434,7 +434,11 @@ pub(crate) mod test {
         // network
 
         let network = dump.network().unwrap().unwrap();
-        insta::assert_snapshot!(network.local.as_ref().unwrap(), @"ms-0");
+
+        // since v1.29 we are dropping `local` and `leader` on import
+        insta::assert_snapshot!(network.local.is_none(), @"true");
+        insta::assert_snapshot!(network.leader.is_none(), @"true");
+
         insta::assert_snapshot!(network.remotes.get("ms-0").as_ref().unwrap().url, @"http://localhost:7700");
         insta::assert_snapshot!(network.remotes.get("ms-0").as_ref().unwrap().search_api_key.is_none(), @"true");
         insta::assert_snapshot!(network.remotes.get("ms-1").as_ref().unwrap().url, @"http://localhost:7701");

crates/dump/src/reader/v2/settings.rs

@@ -107,19 +107,14 @@ impl Settings<Unchecked> {
     }
 }
 
-#[derive(Debug, Clone, PartialEq)]
+#[derive(Default, Debug, Clone, PartialEq)]
 pub enum Setting<T> {
     Set(T),
     Reset,
+    #[default]
     NotSet,
 }
 
-impl<T> Default for Setting<T> {
-    fn default() -> Self {
-        Self::NotSet
-    }
-}
-
 impl<T> Setting<T> {
     pub const fn is_not_set(&self) -> bool {
         matches!(self, Self::NotSet)

crates/dump/src/reader/v3/settings.rs

@@ -161,19 +161,14 @@ pub struct Facets {
     pub min_level_size: Option<NonZeroUsize>,
 }
 
-#[derive(Debug, Clone, PartialEq, Eq)]
+#[derive(Default, Debug, Clone, PartialEq, Eq)]
 pub enum Setting<T> {
     Set(T),
     Reset,
+    #[default]
     NotSet,
 }
 
-impl<T> Default for Setting<T> {
-    fn default() -> Self {
-        Self::NotSet
-    }
-}
-
 impl<T> Setting<T> {
     pub fn map<U, F>(self, f: F) -> Setting<U>
     where

crates/dump/src/reader/v4/meta.rs

@@ -1,9 +1,7 @@
 use std::fmt::{self, Display, Formatter};
-use std::marker::PhantomData;
 use std::str::FromStr;
 
-use serde::de::Visitor;
+use serde::Deserialize;
-use serde::{Deserialize, Deserializer};
 use uuid::Uuid;
 
 use super::settings::{Settings, Unchecked};
@@ -82,59 +80,3 @@ impl Display for IndexUidFormatError {
 }
 
 impl std::error::Error for IndexUidFormatError {}
-
-/// A type that tries to match either a star (*) or
-/// any other thing that implements `FromStr`.
-#[derive(Debug)]
-#[cfg_attr(test, derive(serde::Serialize))]
-pub enum StarOr<T> {
-    Star,
-    Other(T),
-}
-
-impl<'de, T, E> Deserialize<'de> for StarOr<T>
-where
-    T: FromStr<Err = E>,
-    E: Display,
-{
-    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
-    where
-        D: Deserializer<'de>,
-    {
-        /// Serde can't differentiate between `StarOr::Star` and `StarOr::Other` without a tag.
-        /// Simply using `#[serde(untagged)]` + `#[serde(rename="*")]` will lead to attempting to
-        /// deserialize everything as a `StarOr::Other`, including "*".
-        /// [`#[serde(other)]`](https://serde.rs/variant-attrs.html#other) might have helped but is
-        /// not supported on untagged enums.
-        struct StarOrVisitor<T>(PhantomData<T>);
-
-        impl<T, FE> Visitor<'_> for StarOrVisitor<T>
-        where
-            T: FromStr<Err = FE>,
-            FE: Display,
-        {
-            type Value = StarOr<T>;
-
-            fn expecting(&self, formatter: &mut Formatter) -> std::fmt::Result {
-                formatter.write_str("a string")
-            }
-
-            fn visit_str<SE>(self, v: &str) -> Result<Self::Value, SE>
-            where
-                SE: serde::de::Error,
-            {
-                match v {
-                    "*" => Ok(StarOr::Star),
-                    v => {
-                        let other = FromStr::from_str(v).map_err(|e: T::Err| {
-                            SE::custom(format!("Invalid `other` value: {}", e))
-                        })?;
-                        Ok(StarOr::Other(other))
-                    }
-                }
-            }
-        }
-
-        deserializer.deserialize_str(StarOrVisitor(PhantomData))
-    }
-}

crates/dump/src/reader/v4/settings.rs

@@ -192,19 +192,14 @@ pub struct Facets {
     pub min_level_size: Option<NonZeroUsize>,
 }
 
-#[derive(Debug, Clone, PartialEq, Eq, Copy)]
+#[derive(Default, Debug, Clone, PartialEq, Eq, Copy)]
 pub enum Setting<T> {
     Set(T),
     Reset,
+    #[default]
     NotSet,
 }
 
-impl<T> Default for Setting<T> {
-    fn default() -> Self {
-        Self::NotSet
-    }
-}
-
 impl<T> Setting<T> {
     pub fn set(self) -> Option<T> {
         match self {

crates/dump/src/reader/v5/settings.rs

@@ -47,20 +47,15 @@ pub struct Settings<T> {
     pub _kind: PhantomData<T>,
 }
 
-#[derive(Debug, Clone, PartialEq, Eq, Copy)]
+#[derive(Default, Debug, Clone, PartialEq, Eq, Copy)]
 #[cfg_attr(test, derive(serde::Serialize))]
 pub enum Setting<T> {
     Set(T),
     Reset,
+    #[default]
     NotSet,
 }
 
-impl<T> Default for Setting<T> {
-    fn default() -> Self {
-        Self::NotSet
-    }
-}
-
 impl<T> Setting<T> {
     pub fn set(self) -> Option<T> {
         match self {

crates/dump/src/reader/v5/tasks.rs

@@ -322,7 +322,7 @@ impl From<Task> for TaskView {
             _ => None,
         });
 
-        let duration = finished_at.zip(started_at).map(|(tf, ts)| (tf - ts));
+        let duration = finished_at.zip(started_at).map(|(tf, ts)| tf - ts);
 
         Self {
             uid: id,

crates/dump/src/reader/v6/mod.rs

@@ -95,17 +95,26 @@ impl V6Reader {
             Err(e) => return Err(e.into()),
         };
 
-        let network = match fs::read(dump.path().join("network.json")) {
+        let mut network: Option<meilisearch_types::network::Network> =
-            Ok(network_file) => Some(serde_json::from_reader(&*network_file)?),
+            match fs::read(dump.path().join("network.json")) {
-            Err(error) => match error.kind() {
+                Ok(network_file) => Some(serde_json::from_reader(&*network_file)?),
-                // Allows the file to be missing, this will only result in all experimental features disabled.
+                Err(error) => match error.kind() {
-                ErrorKind::NotFound => {
+                    // Allows the file to be missing, this will only result in all experimental features disabled.
-                    debug!("`network.json` not found in dump");
+                    ErrorKind::NotFound => {
-                    None
+                        debug!("`network.json` not found in dump");
-                }
+                        None
-                _ => return Err(error.into()),
+                    }
-            },
+                    _ => return Err(error.into()),
-        };
+                },
+            };
+
+        if let Some(network) = &mut network {
+            // as dumps are typically imported in a different machine as the emitter (otherwise dumpless upgrade would be used),
+            // we decide to remove the self to avoid alias issues
+            network.local = None;
+            // for the same reason we disable automatic sharding
+            network.leader = None;
+        }
 
         let webhooks = match fs::read(dump.path().join("webhooks.json")) {
             Ok(webhooks_file) => Some(serde_json::from_reader(&*webhooks_file)?),

crates/fuzzers/src/bin/fuzz-indexing.rs

@@ -12,7 +12,7 @@ use milli::documents::mmap_from_objects;
 use milli::heed::EnvOpenOptions;
 use milli::progress::Progress;
 use milli::update::new::indexer;
-use milli::update::IndexerConfig;
+use milli::update::{IndexerConfig, MissingDocumentPolicy};
 use milli::vector::RuntimeEmbedders;
 use milli::Index;
 use serde_json::Value;
@@ -113,9 +113,12 @@ fn main() {
 
                             for op in &operations {
                                 match op {
-                                    Either::Left(documents) => {
+                                    Either::Left(documents) => indexer
-                                        indexer.replace_documents(documents).unwrap()
+                                        .replace_documents(
-                                    }
+                                            documents,
+                                            MissingDocumentPolicy::default(),
+                                        )
+                                        .unwrap(),
                                     Either::Right(ids) => indexer.delete_documents(ids),
                                 }
                             }

crates/index-scheduler/Cargo.toml

@@ -24,6 +24,7 @@ dump = { path = "../dump" }
 enum-iterator = "2.3.0"
 file-store = { path = "../file-store" }
 flate2 = "1.1.5"
+hashbrown = "0.15.5"
 indexmap = "2.12.0"
 meilisearch-auth = { path = "../meilisearch-auth" }
 meilisearch-types = { path = "../meilisearch-types" }
@@ -46,10 +47,14 @@ time = { version = "0.3.44", features = [
 tracing = "0.1.41"
 ureq = "2.12.1"
 uuid = { version = "1.18.1", features = ["serde", "v4"] }
-backoff = "0.4.0"
+backoff = { version = "0.4.0", features = ["futures", "tokio"] }
-reqwest = { version = "0.12.24", features = ["rustls-tls", "http2"], default-features = false }
+reqwest = { version = "0.12.24", features = [
+    "rustls-tls",
+    "http2",
+], default-features = false }
 rusty-s3 = "0.8.1"
 tokio = { version = "1.48.0", features = ["full"] }
+urlencoding = "2.1.3"
 
 [dev-dependencies]
 big_s = "1.0.2"
@@ -58,3 +63,6 @@ crossbeam-channel = "0.5.15"
 insta = { version = "=1.39.0", features = ["json", "redactions"] }
 maplit = "1.0.2"
 meili-snap = { path = "../meili-snap" }
+
+[features]
+enterprise = ["meilisearch-types/enterprise"]

crates/index-scheduler/src/dump.rs

@@ -3,17 +3,17 @@
 use std::collections::HashMap;
 use std::io;
 
+use crate::{utils, Error, IndexScheduler, Result};
 use dump::{KindDump, TaskDump, UpdateFile};
 use meilisearch_types::batches::{Batch, BatchId};
 use meilisearch_types::heed::RwTxn;
 use meilisearch_types::index_uid_pattern::IndexUidPattern;
 use meilisearch_types::milli;
+use meilisearch_types::milli::update::MissingDocumentPolicy;
 use meilisearch_types::tasks::{Kind, KindWithContent, Status, Task};
 use roaring::RoaringBitmap;
 use uuid::Uuid;
 
-use crate::{utils, Error, IndexScheduler, Result};
-
 pub struct Dump<'a> {
     index_scheduler: &'a IndexScheduler,
     wtxn: RwTxn<'a>,
@@ -164,6 +164,7 @@ impl<'a> Dump<'a> {
                     content_file: content_uuid.ok_or(Error::CorruptedDump)?,
                     documents_count,
                     allow_index_creation,
+                    on_missing_document: MissingDocumentPolicy::default(),
                 },
                 KindDump::DocumentDeletion { documents_ids } => KindWithContent::DocumentDeletion {
                     documents_ids,
@@ -238,6 +239,9 @@ impl<'a> Dump<'a> {
                 KindDump::IndexCompaction { index_uid } => {
                     KindWithContent::IndexCompaction { index_uid }
                 }
+                KindDump::NetworkTopologyChange(network_topology_change) => {
+                    KindWithContent::NetworkTopologyChange(network_topology_change)
+                }
             },
         };
 

crates/index-scheduler/src/error.rs

@@ -3,10 +3,13 @@ use std::fmt::Display;
 use meilisearch_types::batches::BatchId;
 use meilisearch_types::error::{Code, ErrorCode};
 use meilisearch_types::milli::index::RollbackOutcome;
+use meilisearch_types::milli::DocumentId;
+use meilisearch_types::tasks::network::ReceiveTaskError;
 use meilisearch_types::tasks::{Kind, Status};
 use meilisearch_types::{heed, milli};
 use reqwest::StatusCode;
 use thiserror::Error;
+use uuid::Uuid;
 
 use crate::TaskId;
 
@@ -191,6 +194,17 @@ pub enum Error {
     #[error(transparent)]
     HeedTransaction(heed::Error),
 
+    #[error("No network topology change task is currently enqueued or processing")]
+    ImportTaskWithoutNetworkTask,
+    #[error("The network task version (`{network_task}`) does not match the import task version (`{import_task}`)")]
+    NetworkVersionMismatch { network_task: Uuid, import_task: Uuid },
+    #[error("The import task emanates from an unknown remote `{0}`")]
+    ImportTaskUnknownRemote(String),
+    #[error("The import task with key `{0}` was already received")]
+    ImportTaskAlreadyReceived(DocumentId),
+    #[error("{action} requires the Enterprise Edition")]
+    RequiresEnterpriseEdition { action: &'static str },
+
     #[cfg(test)]
     #[error("Planned failure for tests.")]
     PlannedFailure,
@@ -248,6 +262,11 @@ impl Error {
             | Error::Persist(_)
             | Error::FeatureNotEnabled(_)
             | Error::Export(_)
+            | Error::ImportTaskWithoutNetworkTask
+            | Error::NetworkVersionMismatch { .. }
+            | Error::ImportTaskAlreadyReceived(_)
+            | Error::ImportTaskUnknownRemote(_)
+            | Error::RequiresEnterpriseEdition { .. }
             | Error::Anyhow(_) => true,
             Error::CreateBatch(_)
             | Error::CorruptedTaskQueue
@@ -307,6 +326,11 @@ impl ErrorCode for Error {
             Error::TaskDeletionWithEmptyQuery => Code::MissingTaskFilters,
             Error::TaskCancelationWithEmptyQuery => Code::MissingTaskFilters,
             Error::NoSpaceLeftInTaskQueue => Code::NoSpaceLeftOnDevice,
+            Error::ImportTaskWithoutNetworkTask => Code::ImportTaskWithoutNetworkTask,
+            Error::NetworkVersionMismatch { .. } => Code::NetworkVersionMismatch,
+            Error::ImportTaskAlreadyReceived(_) => Code::ImportTaskAlreadyReceived,
+            Error::ImportTaskUnknownRemote(_) => Code::ImportTaskUnknownRemote,
+            Error::RequiresEnterpriseEdition { .. } => Code::RequiresEnterpriseEdition,
             Error::S3Error { status, .. } if status.is_client_error() => {
                 Code::InvalidS3SnapshotRequest
             }
@@ -345,3 +369,12 @@ impl ErrorCode for Error {
         }
     }
 }
+
+impl From<ReceiveTaskError> for Error {
+    fn from(value: ReceiveTaskError) -> Self {
+        match value {
+            ReceiveTaskError::UnknownRemote(unknown) => Error::ImportTaskUnknownRemote(unknown),
+            ReceiveTaskError::DuplicateTask(dup) => Error::ImportTaskAlreadyReceived(dup),
+        }
+    }
+}

crates/index-scheduler/src/features.rs

@@ -38,6 +38,10 @@ impl RoFeatures {
         Self { runtime }
     }
 
+    pub fn from_runtime_features(features: RuntimeTogglableFeatures) -> Self {
+        Self { runtime: features }
+    }
+
     pub fn runtime_features(&self) -> RuntimeTogglableFeatures {
         self.runtime
     }

crates/index-scheduler/src/index_mapper/mod.rs

@@ -361,6 +361,12 @@ impl IndexMapper {
         Ok(())
     }
 
+    /// The number of indexes in the database
+    #[cfg(feature = "enterprise")] // only used in enterprise edition for now
+    pub fn index_count(&self, rtxn: &RoTxn) -> Result<u64> {
+        Ok(self.index_mapping.len(rtxn)?)
+    }
+
     /// Return an index, may open it if it wasn't already opened.
     pub fn index(&self, rtxn: &RoTxn, name: &str) -> Result<Index> {
         if let Some((current_name, current_index)) =

crates/index-scheduler/src/insta_snapshot.rs

@@ -6,7 +6,7 @@ use meilisearch_types::heed::types::{SerdeBincode, SerdeJson, Str};
 use meilisearch_types::heed::{Database, RoTxn};
 use meilisearch_types::milli::{CboRoaringBitmapCodec, RoaringBitmapCodec, BEU32};
 use meilisearch_types::tasks::{Details, Kind, Status, Task};
-use meilisearch_types::versioning;
+use meilisearch_types::versioning::{self, VERSION_MAJOR, VERSION_MINOR, VERSION_PATCH};
 use roaring::RoaringBitmap;
 
 use crate::index_mapper::IndexMapper;
@@ -27,6 +27,7 @@ pub fn snapshot_index_scheduler(scheduler: &IndexScheduler) -> String {
         queue,
         scheduler,
         persisted,
+        export_default_payload_size_bytes: _,
 
         index_mapper,
         features: _,
@@ -320,11 +321,18 @@ fn snapshot_details(d: &Details) -> String {
             format!("{{ url: {url:?}, api_key: {api_key:?}, payload_size: {payload_size:?}, indexes: {indexes:?} }}")
         }
         Details::UpgradeDatabase { from, to } => {
-            format!("{{ from: {from:?}, to: {to:?} }}")
+            if to == &(VERSION_MAJOR, VERSION_MINOR, VERSION_PATCH) {
+                format!("{{ from: {from:?}, to: [current version] }}")
+            } else {
+                format!("{{ from: {from:?}, to: {to:?} }}")
+            }
         }
         Details::IndexCompaction { index_uid, pre_compaction_size, post_compaction_size } => {
             format!("{{ index_uid: {index_uid:?}, pre_compaction_size: {pre_compaction_size:?}, post_compaction_size: {post_compaction_size:?} }}")
         }
+        Details::NetworkTopologyChange { moved_documents, message } => {
+            format!("{{ moved_documents: {moved_documents:?}, message: {message:?}")
+        }
     }
 }
 
@@ -400,7 +408,21 @@ pub fn snapshot_batch(batch: &Batch) -> String {
 
     snap.push('{');
     snap.push_str(&format!("uid: {uid}, "));
-    snap.push_str(&format!("details: {}, ", serde_json::to_string(details).unwrap()));
+    let details = if let Some(upgrade_to) = &details.upgrade_to {
+        if upgrade_to.as_str()
+            == format!("v{VERSION_MAJOR}.{VERSION_MINOR}.{VERSION_PATCH}").as_str()
+        {
+            let mut details = details.clone();
+
+            details.upgrade_to = Some("[current version]".into());
+            serde_json::to_string(&details).unwrap()
+        } else {
+            serde_json::to_string(details).unwrap()
+        }
+    } else {
+        serde_json::to_string(details).unwrap()
+    };
+    snap.push_str(&format!("details: {details}, "));
     snap.push_str(&format!("stats: {}, ", serde_json::to_string(&stats).unwrap()));
     if !embedder_stats.skip_serializing() {
         snap.push_str(&format!(

crates/index-scheduler/src/lib.rs

@@ -48,6 +48,7 @@ use std::path::{Path, PathBuf};
 use std::sync::{Arc, RwLock};
 use std::time::Duration;
 
+use byte_unit::Byte;
 use dump::Dump;
 pub use error::Error;
 pub use features::RoFeatures;
@@ -68,10 +69,12 @@ use meilisearch_types::milli::vector::{
 use meilisearch_types::milli::{self, Index};
 use meilisearch_types::network::Network;
 use meilisearch_types::task_view::TaskView;
-use meilisearch_types::tasks::{KindWithContent, Task, TaskNetwork};
+use meilisearch_types::tasks::network::{
+    DbTaskNetwork, ImportData, ImportMetadata, Origin, TaskNetwork,
+};
+use meilisearch_types::tasks::{KindWithContent, Task};
 use meilisearch_types::webhooks::{Webhook, WebhooksDumpView, WebhooksView};
 use milli::vector::db::IndexEmbeddingConfig;
-use processing::ProcessingTasks;
 pub use queue::Query;
 use queue::Queue;
 use roaring::RoaringBitmap;
@@ -82,6 +85,7 @@ use uuid::Uuid;
 use versioning::Versioning;
 
 use crate::index_mapper::IndexMapper;
+use crate::processing::ProcessingTasks;
 use crate::utils::clamp_to_page_size;
 
 pub(crate) type BEI128 = I128<BE>;
@@ -144,9 +148,11 @@ pub struct IndexSchedulerOptions {
     /// If the autobatcher is allowed to automatically batch tasks
     /// it will only batch this defined maximum size (in bytes) of tasks at once.
     pub batched_tasks_size_limit: u64,
+    /// The maximum size of the default payload for exporting documents, in bytes
+    pub export_default_payload_size_bytes: Byte,
     /// The experimental features enabled for this instance.
     pub instance_features: InstanceTogglableFeatures,
-    /// The experimental features enabled for this instance.
+    /// Whether the index scheduler is able to auto upgrade or not.
     pub auto_upgrade: bool,
     /// The maximal number of entries in the search query cache of an embedder.
     ///
@@ -199,6 +205,9 @@ pub struct IndexScheduler {
     /// to the same embeddings for the same input text.
     embedders: Arc<RwLock<HashMap<EmbedderOptions, Arc<Embedder>>>>,
 
+    /// The maximum size of the default payload for exporting documents, in bytes
+    pub export_default_payload_size_bytes: Byte,
+
     // ================= test
     // The next entry is dedicated to the tests.
     /// Provide a way to set a breakpoint in multiple part of the scheduler.
@@ -234,6 +243,7 @@ impl IndexScheduler {
             cleanup_enabled: self.cleanup_enabled,
             experimental_no_edition_2024_for_dumps: self.experimental_no_edition_2024_for_dumps,
             persisted: self.persisted,
+            export_default_payload_size_bytes: self.export_default_payload_size_bytes,
 
             webhooks: self.webhooks.clone(),
             embedders: self.embedders.clone(),
@@ -345,6 +355,7 @@ impl IndexScheduler {
             persisted,
             webhooks: Arc::new(webhooks),
             embedders: Default::default(),
+            export_default_payload_size_bytes: options.export_default_payload_size_bytes,
 
             #[cfg(test)] // Will be replaced in `new_tests` in test environments
             test_breakpoint_sdr: crossbeam_channel::bounded(0).0,
@@ -700,14 +711,14 @@ impl IndexScheduler {
         self.queue.get_task_ids_from_authorized_indexes(&rtxn, query, filters, &processing)
     }
 
-    pub fn set_task_network(&self, task_id: TaskId, network: TaskNetwork) -> Result<()> {
+    pub fn set_task_network(&self, task_id: TaskId, network: DbTaskNetwork) -> Result<Task> {
         let mut wtxn = self.env.write_txn()?;
         let mut task =
             self.queue.tasks.get_task(&wtxn, task_id)?.ok_or(Error::TaskNotFound(task_id))?;
         task.network = Some(network);
         self.queue.tasks.all_tasks.put(&mut wtxn, &task_id, &task)?;
         wtxn.commit()?;
-        Ok(())
+        Ok(task)
     }
 
     /// Return the batches matching the query from the user's point of view along
@@ -757,18 +768,30 @@ impl IndexScheduler {
         task_id: Option<TaskId>,
         dry_run: bool,
     ) -> Result<Task> {
-        self.register_with_custom_metadata(kind, task_id, None, dry_run)
+        self.register_with_custom_metadata(kind, task_id, None, dry_run, None)
     }
 
     /// Register a new task in the scheduler, with metadata.
     ///
     /// If it fails and data was associated with the task, it tries to delete the associated data.
+    ///
+    /// # Parameters
+    ///
+    /// - task_network: network of the task to check.
+    ///
+    /// If the task is an import task, only accept it if:
+    ///
+    /// 1. There is an ongoing network topology change task
+    /// 2. The task to register matches the network version of the network topology change task
+    ///
+    /// Always accept the task if it is not an import task.
     pub fn register_with_custom_metadata(
         &self,
         kind: KindWithContent,
         task_id: Option<TaskId>,
         custom_metadata: Option<String>,
         dry_run: bool,
+        task_network: Option<TaskNetwork>,
     ) -> Result<Task> {
         // if the task doesn't delete or cancel anything and 40% of the task queue is full, we must refuse to enqueue the incoming task
         if !matches!(&kind, KindWithContent::TaskDeletion { tasks, .. } | KindWithContent::TaskCancelation { tasks, .. } if !tasks.is_empty())
@@ -779,7 +802,19 @@ impl IndexScheduler {
         }
 
         let mut wtxn = self.env.write_txn()?;
-        let task = self.queue.register(&mut wtxn, &kind, task_id, custom_metadata, dry_run)?;
+
+        if let Some(TaskNetwork::Import { import_from, network_change, metadata }) = &task_network {
+            self.update_network_task(&mut wtxn, import_from, network_change, metadata)?;
+        }
+
+        let task = self.queue.register(
+            &mut wtxn,
+            &kind,
+            task_id,
+            custom_metadata,
+            dry_run,
+            task_network.map(DbTaskNetwork::from),
+        )?;
 
         // If the registered task is a task cancelation
         // we inform the processing tasks to stop (if necessary).
@@ -801,6 +836,91 @@ impl IndexScheduler {
         Ok(task)
     }
 
+    pub fn network_no_index_for_remote(
+        &self,
+        remote_name: String,
+        origin: Origin,
+    ) -> Result<(), Error> {
+        let mut wtxn = self.env.write_txn()?;
+
+        self.update_network_task(
+            &mut wtxn,
+            &ImportData { remote_name, index_name: None, document_count: 0 },
+            &origin,
+            &ImportMetadata { index_count: 0, task_key: None, total_index_documents: 0 },
+        )?;
+
+        wtxn.commit()?;
+
+        // wake up the scheduler as the task state has changed
+        self.scheduler.wake_up.signal();
+
+        Ok(())
+    }
+
+    fn update_network_task(
+        &self,
+        wtxn: &mut heed::RwTxn<'_>,
+        import_from: &ImportData,
+        network_change: &Origin,
+        metadata: &ImportMetadata,
+    ) -> Result<(), Error> {
+        let mut network_tasks = self
+            .queue
+            .tasks
+            .get_kind(&*wtxn, meilisearch_types::tasks::Kind::NetworkTopologyChange)?;
+        if network_tasks.is_empty() {
+            return Err(Error::ImportTaskWithoutNetworkTask);
+        }
+        let network_task = {
+            let processing = self.processing_tasks.read().unwrap().processing.clone();
+            if processing.is_disjoint(&network_tasks) {
+                let enqueued = self
+                    .queue
+                    .tasks
+                    .get_status(&*wtxn, meilisearch_types::tasks::Status::Enqueued)?;
+
+                network_tasks &= enqueued;
+                if let Some(network_task) = network_tasks.into_iter().next() {
+                    network_task
+                } else {
+                    return Err(Error::ImportTaskWithoutNetworkTask);
+                }
+            } else {
+                network_tasks &= &*processing;
+                network_tasks.into_iter().next().unwrap()
+            }
+        };
+        let mut network_task = self.queue.tasks.get_task(&*wtxn, network_task)?.unwrap();
+        let network_task_version = network_task
+            .network
+            .as_ref()
+            .map(|network| network.network_version())
+            .unwrap_or_default();
+        if network_task_version != network_change.network_version {
+            return Err(Error::NetworkVersionMismatch {
+                network_task: network_task_version,
+                import_task: network_change.network_version,
+            });
+        }
+        let KindWithContent::NetworkTopologyChange(network_topology_change) =
+            &mut network_task.kind
+        else {
+            tracing::error!("unexpected network kind for network task while registering task");
+            return Err(Error::CorruptedTaskQueue);
+        };
+        network_topology_change.receive_remote_task(
+            &import_from.remote_name,
+            import_from.index_name.as_deref(),
+            metadata.task_key,
+            import_from.document_count,
+            metadata.index_count,
+            metadata.total_index_documents,
+        )?;
+        self.queue.tasks.update_task(wtxn, &mut network_task)?;
+        Ok(())
+    }
+
     /// Register a new task coming from a dump in the scheduler.
     /// By taking a mutable ref we're pretty sure no one will ever import a dump while actix is running.
     pub fn register_dumped_task(&mut self) -> Result<Dump<'_>> {

crates/index-scheduler/src/processing.rs

@@ -42,12 +42,10 @@ impl ProcessingTasks {
 
     /// Set the processing tasks to an empty list
     pub fn stop_processing(&mut self) -> Self {
-        self.progress = None;
-
         Self {
             batch: std::mem::take(&mut self.batch),
             processing: std::mem::take(&mut self.processing),
-            progress: None,
+            progress: std::mem::take(&mut self.progress),
         }
     }
 

crates/index-scheduler/src/queue/batches_test.rs

@@ -1,14 +1,17 @@
+use crate::insta_snapshot::{snapshot_bitmap, snapshot_index_scheduler};
+use crate::test_utils::Breakpoint::*;
+use crate::test_utils::{
+    index_creation_task, replace_document_import_task, replace_document_import_task_with_opts,
+    sample_documents, FailureLocation,
+};
+use crate::{IndexScheduler, Query};
 use meili_snap::snapshot;
 use meilisearch_auth::AuthFilter;
 use meilisearch_types::index_uid_pattern::IndexUidPattern;
+use meilisearch_types::milli::update::MissingDocumentPolicy;
 use meilisearch_types::tasks::{IndexSwap, KindWithContent, Status};
 use time::{Duration, OffsetDateTime};
 
-use crate::insta_snapshot::{snapshot_bitmap, snapshot_index_scheduler};
-use crate::test_utils::Breakpoint::*;
-use crate::test_utils::{index_creation_task, FailureLocation};
-use crate::{IndexScheduler, Query};
-
 #[test]
 fn query_batches_from_and_limit() {
     let (index_scheduler, mut handle) = IndexScheduler::test(true, vec![]);
@@ -487,3 +490,41 @@ fn query_batches_canceled_by() {
     // Return only 1 because the user is not authorized to see task 2
     snapshot!(snapshot_bitmap(&batches), @"[1,]");
 }
+
+#[test]
+fn batch_skip_creation_with_deletion() {
+    let (index_scheduler, mut handle) = IndexScheduler::test(true, vec![]);
+    let kind = index_creation_task("docs", "id");
+    let _task = index_scheduler.register(kind, None, false).unwrap();
+
+    handle.advance_one_successful_batch();
+
+    let (file0, documents_count0) = sample_documents(&index_scheduler, 1, 1);
+    let (file1, documents_count1) = sample_documents(&index_scheduler, 2, 1);
+    file0.persist().unwrap();
+    file1.persist().unwrap();
+    let kind = replace_document_import_task("docs", Some("id"), 1, documents_count0);
+    index_scheduler.register(kind, None, false).unwrap();
+    index_scheduler
+        .register(
+            KindWithContent::DocumentDeletion {
+                index_uid: "docs".to_string(),
+                documents_ids: vec!["1".to_string()],
+            },
+            None,
+            false,
+        )
+        .unwrap();
+    let kind = replace_document_import_task_with_opts(
+        "docs",
+        Some("id"),
+        2,
+        documents_count1,
+        MissingDocumentPolicy::Skip,
+    );
+    index_scheduler.register(kind, None, false).unwrap();
+
+    handle.advance_one_successful_batch();
+
+    snapshot!(snapshot_index_scheduler(&index_scheduler));
+}

crates/index-scheduler/src/queue/mod.rs

@@ -15,6 +15,7 @@ use file_store::FileStore;
 use meilisearch_types::batches::BatchId;
 use meilisearch_types::heed::{Database, Env, RoTxn, RwTxn, WithoutTls};
 use meilisearch_types::milli::{CboRoaringBitmapCodec, BEU32};
+use meilisearch_types::tasks::network::DbTaskNetwork;
 use meilisearch_types::tasks::{Kind, KindWithContent, Status, Task};
 use roaring::RoaringBitmap;
 use time::format_description::well_known::Rfc3339;
@@ -259,6 +260,7 @@ impl Queue {
         task_id: Option<TaskId>,
         custom_metadata: Option<String>,
         dry_run: bool,
+        network: Option<DbTaskNetwork>,
     ) -> Result<Task> {
         let next_task_id = self.tasks.next_task_id(wtxn)?;
 
@@ -280,7 +282,7 @@ impl Queue {
             details: kind.default_details(),
             status: Status::Enqueued,
             kind: kind.clone(),
-            network: None,
+            network,
             custom_metadata,
         };
         // For deletion and cancelation tasks, we want to make extra sure that they
@@ -348,6 +350,7 @@ impl Queue {
             None,
             None,
             false,
+            None,
         )?;
 
         Ok(())

crates/index-scheduler/src/queue/snapshots/batches_test.rs/batch_skip_creation_with_deletion/1.snap

@@ -0,0 +1,81 @@
+---
+source: crates/index-scheduler/src/queue/batches_test.rs
+---
+### Autobatching Enabled = true
+### Processing batch None:
+[]
+----------------------------------------------------------------------
+### All Tasks:
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("id"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "docs", primary_key: Some("id") }}
+1 {uid: 1, batch_uid: 1, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "docs", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
+2 {uid: 2, batch_uid: 1, status: succeeded, details: { received_document_ids: 1, deleted_documents: Some(1) }, kind: DocumentDeletion { index_uid: "docs", documents_ids: ["1"] }}
+3 {uid: 3, batch_uid: 1, status: succeeded, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "docs", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 1, allow_index_creation: true, on_missing_document: Skip }}
+----------------------------------------------------------------------
+### Status:
+enqueued []
+succeeded [0,1,2,3,]
+----------------------------------------------------------------------
+### Kind:
+"documentAdditionOrUpdate" [1,3,]
+"documentDeletion" [2,]
+"indexCreation" [0,]
+----------------------------------------------------------------------
+### Index Tasks:
+docs [0,1,2,3,]
+----------------------------------------------------------------------
+### Index Mapper:
+docs: { number_of_documents: 0, field_distribution: {} }
+
+----------------------------------------------------------------------
+### Canceled By:
+
+----------------------------------------------------------------------
+### Enqueued At:
+[timestamp] [0,]
+[timestamp] [1,]
+[timestamp] [2,]
+[timestamp] [3,]
+----------------------------------------------------------------------
+### Started At:
+[timestamp] [0,]
+[timestamp] [1,2,3,]
+----------------------------------------------------------------------
+### Finished At:
+[timestamp] [0,]
+[timestamp] [1,2,3,]
+----------------------------------------------------------------------
+### All Batches:
+0 {uid: 0, details: {"primaryKey":"id"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"docs":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
+1 {uid: 1, details: {"receivedDocuments":2,"indexedDocuments":1,"providedIds":1,"deletedDocuments":1}, stats: {"totalNbTasks":3,"status":{"succeeded":3},"types":{"documentAdditionOrUpdate":2,"documentDeletion":1},"indexUids":{"docs":3}}, stop reason: "batched all enqueued tasks", }
+----------------------------------------------------------------------
+### Batch to tasks mapping:
+0 [0,]
+1 [1,2,3,]
+----------------------------------------------------------------------
+### Batches Status:
+succeeded [0,1,]
+----------------------------------------------------------------------
+### Batches Kind:
+"documentAdditionOrUpdate" [1,]
+"documentDeletion" [1,]
+"indexCreation" [0,]
+----------------------------------------------------------------------
+### Batches Index Tasks:
+docs [0,1,]
+----------------------------------------------------------------------
+### Batches Enqueued At:
+[timestamp] [0,]
+[timestamp] [1,]
+[timestamp] [1,]
+----------------------------------------------------------------------
+### Batches Started At:
+[timestamp] [0,]
+[timestamp] [1,]
+----------------------------------------------------------------------
+### Batches Finished At:
+[timestamp] [0,]
+[timestamp] [1,]
+----------------------------------------------------------------------
+### File Store:
+
+----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/test.rs/register/everything_is_successfully_registered.snap

@@ -7,9 +7,9 @@ source: crates/index-scheduler/src/queue/test.rs
 ----------------------------------------------------------------------
 ### All Tasks:
 0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { received_documents: 12, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 12, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 12, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 12, allow_index_creation: true, on_missing_document: Create }}
-2 {uid: 2, status: enqueued, details: { received_documents: 50, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 50, allow_index_creation: true }}
+2 {uid: 2, status: enqueued, details: { received_documents: 50, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 50, allow_index_creation: true, on_missing_document: Create }}
-3 {uid: 3, status: enqueued, details: { received_documents: 5000, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 5000, allow_index_creation: true }}
+3 {uid: 3, status: enqueued, details: { received_documents: 5000, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 5000, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,2,3,]

crates/index-scheduler/src/queue/tasks.rs

@@ -3,7 +3,8 @@ use std::ops::{Bound, RangeBounds};
 use meilisearch_types::heed::types::{DecodeIgnore, SerdeBincode, SerdeJson, Str};
 use meilisearch_types::heed::{Database, Env, RoTxn, RwTxn, WithoutTls};
 use meilisearch_types::milli::{CboRoaringBitmapCodec, RoaringBitmapCodec, BEU32};
-use meilisearch_types::tasks::{Kind, Status, Task};
+use meilisearch_types::tasks::network::DbTaskNetwork;
+use meilisearch_types::tasks::{Kind, KindWithContent, Status, Task};
 use roaring::{MultiOps, RoaringBitmap};
 use time::OffsetDateTime;
 
@@ -114,14 +115,15 @@ impl TaskQueue {
     /// - CorruptedTaskQueue: The task doesn't exist in the database
     pub(crate) fn update_task(&self, wtxn: &mut RwTxn, task: &mut Task) -> Result<()> {
         let old_task = self.get_task(wtxn, task.uid)?.ok_or(Error::CorruptedTaskQueue)?;
-        let reprocessing = old_task.status != Status::Enqueued;
+        // network topology tasks may be processed multiple times.
+        let maybe_reprocessing = old_task.status != Status::Enqueued
+            || task.kind.as_kind() == Kind::NetworkTopologyChange;
 
-        debug_assert!(old_task != *task);
         debug_assert_eq!(old_task.uid, task.uid);
 
         // If we're processing a task that failed it may already contains a batch_uid
         debug_assert!(
-            reprocessing || (old_task.batch_uid.is_none() && task.batch_uid.is_some()),
+            maybe_reprocessing || (old_task.batch_uid.is_none() && task.batch_uid.is_some()),
             "\n==> old: {old_task:?}\n==> new: {task:?}"
         );
 
@@ -143,13 +145,24 @@ impl TaskQueue {
             })?;
         }
 
+        // Avoids rewriting part of the network topology change because of TOCTOU errors
+        if let (
+            KindWithContent::NetworkTopologyChange(old_state),
+            KindWithContent::NetworkTopologyChange(new_state),
+        ) = (old_task.kind, &mut task.kind)
+        {
+            new_state.merge(old_state);
+            // the state possibly just changed, rewrite the details
+            task.details = Some(new_state.to_details());
+        }
+
         assert_eq!(
             old_task.enqueued_at, task.enqueued_at,
             "Cannot update a task's enqueued_at time"
         );
         if old_task.started_at != task.started_at {
             assert!(
-                reprocessing || old_task.started_at.is_none(),
+                maybe_reprocessing || old_task.started_at.is_none(),
                 "Cannot update a task's started_at time"
             );
             if let Some(started_at) = old_task.started_at {
@@ -161,7 +174,7 @@ impl TaskQueue {
         }
         if old_task.finished_at != task.finished_at {
             assert!(
-                reprocessing || old_task.finished_at.is_none(),
+                maybe_reprocessing || old_task.finished_at.is_none(),
                 "Cannot update a task's finished_at time"
             );
             if let Some(finished_at) = old_task.finished_at {
@@ -175,7 +188,16 @@ impl TaskQueue {
         task.network = match (old_task.network, task.network.take()) {
             (None, None) => None,
             (None, Some(network)) | (Some(network), None) => Some(network),
-            (Some(_), Some(network)) => Some(network),
+            (Some(left), Some(right)) => Some(match (left, right) {
+                (
+                    DbTaskNetwork::Remotes { remote_tasks: mut left, network_version: _ },
+                    DbTaskNetwork::Remotes { remote_tasks: mut right, network_version },
+                ) => {
+                    left.append(&mut right);
+                    DbTaskNetwork::Remotes { remote_tasks: left, network_version }
+                }
+                (_, right) => right,
+            }),
         };
 
         self.all_tasks.put(wtxn, &task.uid, task)?;

crates/index-scheduler/src/queue/test.rs

@@ -203,26 +203,30 @@ fn test_disable_auto_deletion_of_tasks() {
         )
         .unwrap();
 
-    let rtxn = index_scheduler.env.read_txn().unwrap();
+    {
-    let proc = index_scheduler.processing_tasks.read().unwrap();
+        let rtxn = index_scheduler.env.read_txn().unwrap();
-    let tasks =
+        let proc = index_scheduler.processing_tasks.read().unwrap();
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
+        let tasks = index_scheduler
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+            .queue
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]" }), name: "task_queue_is_full");
+            .get_task_ids(&rtxn, &Query { ..Default::default() }, &proc)
-    drop(rtxn);
+            .unwrap();
-    drop(proc);
+        let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+        snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]" }), name: "task_queue_is_full");
+    }
 
     // now we're above the max number of tasks
     // and if we try to advance in the tick function no new task deletion should be enqueued
     handle.advance_till([Start, BatchCreated]);
-    let rtxn = index_scheduler.env.read_txn().unwrap();
+    {
-    let proc = index_scheduler.processing_tasks.read().unwrap();
+        let rtxn = index_scheduler.env.read_txn().unwrap();
-    let tasks =
+        let proc = index_scheduler.processing_tasks.read().unwrap();
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
+        let tasks = index_scheduler
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+            .queue
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "task_deletion_have_not_been_enqueued");
+            .get_task_ids(&rtxn, &Query { ..Default::default() }, &proc)
-    drop(rtxn);
+            .unwrap();
-    drop(proc);
+        let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+        snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "task_deletion_have_not_been_enqueued");
+    }
 }
 
 #[test]
@@ -267,59 +271,69 @@ fn test_auto_deletion_of_tasks() {
         )
         .unwrap();
 
-    let rtxn = index_scheduler.env.read_txn().unwrap();
+    {
-    let proc = index_scheduler.processing_tasks.read().unwrap();
+        let rtxn = index_scheduler.env.read_txn().unwrap();
-    let tasks =
+        let proc = index_scheduler.processing_tasks.read().unwrap();
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
+        let tasks = index_scheduler
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+            .queue
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]" }), name: "task_queue_is_full");
+            .get_task_ids(&rtxn, &Query { ..Default::default() }, &proc)
-    drop(rtxn);
+            .unwrap();
-    drop(proc);
+        let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+        snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]" }), name: "task_queue_is_full");
+    }
 
-    // now we're above the max number of tasks
+    {
-    // and if we try to advance in the tick function a new task deletion should be enqueued
+        // now we're above the max number of tasks
-    handle.advance_till([Start, BatchCreated]);
+        // and if we try to advance in the tick function a new task deletion should be enqueued
-    let rtxn = index_scheduler.env.read_txn().unwrap();
+        handle.advance_till([Start, BatchCreated]);
-    let proc = index_scheduler.processing_tasks.read().unwrap();
+        let rtxn = index_scheduler.env.read_txn().unwrap();
-    let tasks =
+        let proc = index_scheduler.processing_tasks.read().unwrap();
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
+        let tasks = index_scheduler
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+            .queue
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "task_deletion_have_been_enqueued");
+            .get_task_ids(&rtxn, &Query { ..Default::default() }, &proc)
-    drop(rtxn);
+            .unwrap();
-    drop(proc);
+        let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+        snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "task_deletion_have_been_enqueued");
+    }
 
-    handle.advance_till([InsideProcessBatch, ProcessBatchSucceeded, AfterProcessing]);
+    {
-    let rtxn = index_scheduler.env.read_txn().unwrap();
+        handle.advance_till([InsideProcessBatch, ProcessBatchSucceeded, AfterProcessing]);
-    let proc = index_scheduler.processing_tasks.read().unwrap();
+        let rtxn = index_scheduler.env.read_txn().unwrap();
-    let tasks =
+        let proc = index_scheduler.processing_tasks.read().unwrap();
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
+        let tasks = index_scheduler
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+            .queue
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "task_deletion_have_been_processed");
+            .get_task_ids(&rtxn, &Query { ..Default::default() }, &proc)
-    drop(rtxn);
+            .unwrap();
-    drop(proc);
+        let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+        snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "task_deletion_have_been_processed");
+    }
 
     handle.advance_one_failed_batch();
     // a new task deletion has been enqueued
     handle.advance_one_successful_batch();
-    let rtxn = index_scheduler.env.read_txn().unwrap();
+    {
-    let proc = index_scheduler.processing_tasks.read().unwrap();
+        let rtxn = index_scheduler.env.read_txn().unwrap();
-    let tasks =
+        let proc = index_scheduler.processing_tasks.read().unwrap();
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
+        let tasks = index_scheduler
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+            .queue
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "after_the_second_task_deletion");
+            .get_task_ids(&rtxn, &Query { ..Default::default() }, &proc)
-    drop(rtxn);
+            .unwrap();
-    drop(proc);
+        let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+        snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "after_the_second_task_deletion");
+    }
 
     handle.advance_one_failed_batch();
     handle.advance_one_successful_batch();
-    let rtxn = index_scheduler.env.read_txn().unwrap();
+    {
-    let proc = index_scheduler.processing_tasks.read().unwrap();
+        let rtxn = index_scheduler.env.read_txn().unwrap();
-    let tasks =
+        let proc = index_scheduler.processing_tasks.read().unwrap();
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
+        let tasks = index_scheduler
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+            .queue
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "everything_has_been_processed");
+            .get_task_ids(&rtxn, &Query { ..Default::default() }, &proc)
-    drop(rtxn);
+            .unwrap();
-    drop(proc);
+        let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
+        snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "everything_has_been_processed");
+    }
 }
 
 #[test]

crates/index-scheduler/src/scheduler/autobatcher.rs

@@ -74,6 +74,7 @@ impl From<KindWithContent> for AutobatchKind {
             | KindWithContent::DumpCreation { .. }
             | KindWithContent::Export { .. }
             | KindWithContent::UpgradeDatabase { .. }
+            | KindWithContent::NetworkTopologyChange(_)
             | KindWithContent::SnapshotCreation => {
                 panic!("The autobatcher should never be called with tasks with special priority or that don't apply to an index.")
             }

crates/index-scheduler/src/scheduler/autobatcher_test.rs

@@ -1,12 +1,12 @@
-use meilisearch_types::milli::update::IndexDocumentsMethod::{
-    self, ReplaceDocuments, UpdateDocuments,
-};
-use meilisearch_types::tasks::{BatchStopReason, IndexSwap, KindWithContent};
-use uuid::Uuid;
-
 use self::autobatcher::{autobatch, BatchKind};
 use super::*;
 use crate::TaskId;
+use meilisearch_types::milli::update::IndexDocumentsMethod::{
+    self, ReplaceDocuments, UpdateDocuments,
+};
+use meilisearch_types::milli::update::MissingDocumentPolicy;
+use meilisearch_types::tasks::{BatchStopReason, IndexSwap, KindWithContent};
+use uuid::Uuid;
 
 #[macro_export]
 macro_rules! debug_snapshot {
@@ -40,6 +40,7 @@ fn doc_imp(
         content_file: Uuid::new_v4(),
         documents_count: 0,
         allow_index_creation,
+        on_missing_document: MissingDocumentPolicy::default(),
     }
 }
 

crates/index-scheduler/src/scheduler/community_edition.rs

@@ -0,0 +1,37 @@
+use meilisearch_types::milli::progress::Progress;
+use meilisearch_types::tasks::Task;
+
+use super::create_batch::Batch;
+use crate::scheduler::process_batch::ProcessBatchInfo;
+use crate::utils::ProcessingBatch;
+use crate::{Error, IndexScheduler, Result};
+
+impl IndexScheduler {
+    pub(super) fn process_network_index_batch(
+        &self,
+        _network_task: Task,
+        _inner_batch: Box<Batch>,
+        _current_batch: &mut ProcessingBatch,
+        _progress: Progress,
+    ) -> Result<(Vec<Task>, ProcessBatchInfo)> {
+        Err(Error::RequiresEnterpriseEdition { action: "processing a network task" })
+    }
+
+    pub(super) fn process_network_ready(
+        &self,
+        _task: Task,
+        _progress: Progress,
+    ) -> Result<(Vec<Task>, ProcessBatchInfo)> {
+        Err(Error::RequiresEnterpriseEdition { action: "processing a network task" })
+    }
+
+    #[cfg(unix)]
+    pub(super) async fn process_snapshot_to_s3(
+        &self,
+        _progress: Progress,
+        _opts: meilisearch_types::milli::update::S3SnapshotOptions,
+        _tasks: Vec<Task>,
+    ) -> Result<Vec<Task>> {
+        Err(Error::RequiresEnterpriseEdition { action: "processing an S3-streaming snapshot task" })
+    }
+}

crates/index-scheduler/src/scheduler/create_batch.rs

@@ -2,8 +2,9 @@ use std::fmt;
 use std::io::ErrorKind;
 
 use meilisearch_types::heed::RoTxn;
-use meilisearch_types::milli::update::IndexDocumentsMethod;
+use meilisearch_types::milli::update::{IndexDocumentsMethod, MissingDocumentPolicy};
 use meilisearch_types::settings::{Settings, Unchecked};
+use meilisearch_types::tasks::network::NetworkTopologyState;
 use meilisearch_types::tasks::{BatchStopReason, Kind, KindWithContent, Status, Task};
 use roaring::RoaringBitmap;
 use uuid::Uuid;
@@ -59,12 +60,20 @@ pub(crate) enum Batch {
         index_uid: String,
         task: Task,
     },
+    #[allow(clippy::enum_variant_names)] // warranted because we are executing an inner index batch
+    NetworkIndexBatch {
+        network_task: Task,
+        inner_batch: Box<Batch>,
+    },
+    NetworkReady {
+        task: Task,
+    },
 }
 
 #[derive(Debug)]
 pub(crate) enum DocumentOperation {
-    Replace(Uuid),
+    Replace { content_file: Uuid, on_missing_document: MissingDocumentPolicy },
-    Update(Uuid),
+    Update { content_file: Uuid, on_missing_document: MissingDocumentPolicy },
     Delete(Vec<String>),
 }
 
@@ -140,9 +149,14 @@ impl Batch {
                     ..
                 } => RoaringBitmap::from_iter(tasks.iter().chain(other).map(|task| task.uid)),
             },
-            Batch::IndexSwap { task } => {
+            Batch::IndexSwap { task } | Batch::NetworkReady { task } => {
                 RoaringBitmap::from_sorted_iter(std::iter::once(task.uid)).unwrap()
             }
+            Batch::NetworkIndexBatch { network_task, inner_batch } => {
+                let mut tasks = inner_batch.ids();
+                tasks.insert(network_task.uid);
+                tasks
+            }
         }
     }
 
@@ -156,12 +170,14 @@ impl Batch {
             | Dump(_)
             | Export { .. }
             | UpgradeDatabase { .. }
+            | NetworkReady { .. }
             | IndexSwap { .. } => None,
             IndexOperation { op, .. } => Some(op.index_uid()),
             IndexCreation { index_uid, .. }
             | IndexUpdate { index_uid, .. }
             | IndexDeletion { index_uid, .. }
             | IndexCompaction { index_uid, .. } => Some(index_uid),
+            NetworkIndexBatch { network_task: _, inner_batch } => inner_batch.index_uid(),
         }
     }
 }
@@ -184,6 +200,8 @@ impl fmt::Display for Batch {
             Batch::IndexCompaction { .. } => f.write_str("IndexCompaction")?,
             Batch::Export { .. } => f.write_str("Export")?,
             Batch::UpgradeDatabase { .. } => f.write_str("UpgradeDatabase")?,
+            Batch::NetworkIndexBatch { .. } => f.write_str("NetworkTopologyChange")?,
+            Batch::NetworkReady { .. } => f.write_str("NetworkTopologyChange")?,
         };
         match index_uid {
             Some(name) => f.write_fmt(format_args!(" on {name:?} from tasks: {tasks:?}")),
@@ -293,13 +311,22 @@ impl IndexScheduler {
                 for task in tasks.iter() {
                     match task.kind {
                         KindWithContent::DocumentAdditionOrUpdate {
-                            content_file, method, ..
+                            content_file,
+                            method,
+                            on_missing_document,
+                            ..
                         } => match method {
                             IndexDocumentsMethod::ReplaceDocuments => {
-                                operations.push(DocumentOperation::Replace(content_file))
+                                operations.push(DocumentOperation::Replace {
+                                    content_file,
+                                    on_missing_document,
+                                })
                             }
                             IndexDocumentsMethod::UpdateDocuments => {
-                                operations.push(DocumentOperation::Update(content_file))
+                                operations.push(DocumentOperation::Update {
+                                    content_file,
+                                    on_missing_document,
+                                })
                             }
                             _ => unreachable!("Unknown document merging method"),
                         },
@@ -452,6 +479,7 @@ impl IndexScheduler {
     pub(crate) fn create_next_batch(
         &self,
         rtxn: &RoTxn,
+        processing_network_tasks: &RoaringBitmap,
     ) -> Result<Option<(Batch, ProcessingBatch)>> {
         #[cfg(test)]
         self.maybe_fail(crate::test_utils::FailureLocation::InsideCreateBatch)?;
@@ -460,7 +488,6 @@ impl IndexScheduler {
         let mut current_batch = ProcessingBatch::new(batch_id);
 
         let enqueued = &self.queue.tasks.get_status(rtxn, Status::Enqueued)?;
-        let count_total_enqueued = enqueued.len();
         let failed = &self.queue.tasks.get_status(rtxn, Status::Failed)?;
 
         // 0. we get the last task to cancel.
@@ -509,7 +536,15 @@ impl IndexScheduler {
             )));
         }
 
-        // 2. we get the next task to delete
+        // 2. Check for enqueued network topology changes
+        let network_changes = self.queue.tasks.get_kind(rtxn, Kind::NetworkTopologyChange)?
+            & (enqueued | processing_network_tasks);
+        if let Some(task_id) = network_changes.iter().next() {
+            let task = self.queue.tasks.get_task(rtxn, task_id)?.unwrap();
+            return self.start_processing_network(rtxn, task, enqueued, current_batch);
+        }
+
+        // 3. we get the next task to delete
         let to_delete = self.queue.tasks.get_kind(rtxn, Kind::TaskDeletion)? & enqueued;
         if !to_delete.is_empty() {
             let mut tasks = self.queue.tasks.get_existing_tasks(rtxn, to_delete)?;
@@ -519,7 +554,7 @@ impl IndexScheduler {
             return Ok(Some((Batch::TaskDeletions(tasks), current_batch)));
         }
 
-        // 3. we get the next task to compact
+        // 4. we get the next task to compact
         let to_compact = self.queue.tasks.get_kind(rtxn, Kind::IndexCompaction)? & enqueued;
         if let Some(task_id) = to_compact.min() {
             let mut task =
@@ -534,7 +569,7 @@ impl IndexScheduler {
             return Ok(Some((Batch::IndexCompaction { index_uid, task }, current_batch)));
         }
 
-        // 4. we batch the export.
+        // 5. we batch the export.
         let to_export = self.queue.tasks.get_kind(rtxn, Kind::Export)? & enqueued;
         if !to_export.is_empty() {
             let task_id = to_export.iter().next().expect("There must be at least one export task");
@@ -545,7 +580,7 @@ impl IndexScheduler {
             return Ok(Some((Batch::Export { task }, current_batch)));
         }
 
-        // 5. we batch the snapshot.
+        // 6. we batch the snapshot.
         let to_snapshot = self.queue.tasks.get_kind(rtxn, Kind::SnapshotCreation)? & enqueued;
         if !to_snapshot.is_empty() {
             let mut tasks = self.queue.tasks.get_existing_tasks(rtxn, to_snapshot)?;
@@ -555,7 +590,7 @@ impl IndexScheduler {
             return Ok(Some((Batch::SnapshotCreation(tasks), current_batch)));
         }
 
-        // 6. we batch the dumps.
+        // 7. we batch the dumps.
         let to_dump = self.queue.tasks.get_kind(rtxn, Kind::DumpCreation)? & enqueued;
         if let Some(to_dump) = to_dump.min() {
             let mut task =
@@ -568,25 +603,66 @@ impl IndexScheduler {
             return Ok(Some((Batch::Dump(task), current_batch)));
         }
 
-        // 7. We make a batch from the unprioritised tasks. Start by taking the next enqueued task.
+        let network = self.network();
-        let task_id = if let Some(task_id) = enqueued.min() { task_id } else { return Ok(None) };
-        let mut task =
-            self.queue.tasks.get_task(rtxn, task_id)?.ok_or(Error::CorruptedTaskQueue)?;
 
-        // If the task is not associated with any index, verify that it is an index swap and
+        // 8. We make a batch from the unprioritised tasks.
-        // create the batch directly. Otherwise, get the index name associated with the task
+        let (batch, current_batch) =
-        // and use the autobatcher to batch the enqueued tasks associated with it
+            self.create_next_batch_unprioritized(rtxn, enqueued, current_batch, |task| {
+                // We want to execute all tasks, except those that have a version strictly higher than the network version
 
-        let index_name = if let Some(&index_name) = task.indexes().first() {
+                let Some(task_version) =
-            index_name
+                    task.network.as_ref().map(|tastk_network| tastk_network.network_version())
-        } else {
+                else {
-            assert!(matches!(&task.kind, KindWithContent::IndexSwap { swaps } if swaps.is_empty()));
+                    // do not skip tasks that have no network version, otherwise we will never execute them
-            current_batch.processing(Some(&mut task));
+                    return false;
-            current_batch.reason(BatchStopReason::TaskCannotBeBatched {
+                };
-                kind: Kind::IndexSwap,
+
-                id: task.uid,
+                // skip tasks with a version strictly higher than the network version
-            });
+                task_version > network.version
-            return Ok(Some((Batch::IndexSwap { task }, current_batch)));
+            })?;
+        Ok(batch.map(|batch| (batch, current_batch)))
+    }
+
+    fn create_next_batch_unprioritized<F>(
+        &self,
+        rtxn: &RoTxn,
+        enqueued: &RoaringBitmap,
+        mut current_batch: ProcessingBatch,
+        mut skip_if: F,
+    ) -> Result<(Option<Batch>, ProcessingBatch)>
+    where
+        F: FnMut(&Task) -> bool,
+    {
+        let count_total_enqueued = enqueued.len();
+
+        let mut enqueued_it = enqueued.iter();
+        let mut task;
+        let index_name = loop {
+            let Some(task_id) = enqueued_it.next() else {
+                return Ok((None, current_batch));
+            };
+            task = self.queue.tasks.get_task(rtxn, task_id)?.ok_or(Error::CorruptedTaskQueue)?;
+
+            if skip_if(&task) {
+                continue;
+            }
+            // If the task is not associated with any index, verify that it is an index swap and
+            // create the batch directly. Otherwise, get the index name associated with the task
+            // and use the autobatcher to batch the enqueued tasks associated with it
+
+            if let Some(&index_name) = task.indexes().first() {
+                break index_name;
+            } else {
+                assert!(
+                    matches!(&task.kind, KindWithContent::IndexSwap { swaps } if swaps.is_empty())
+                );
+                current_batch.processing(Some(&mut task));
+                current_batch.reason(BatchStopReason::TaskCannotBeBatched {
+                    kind: Kind::IndexSwap,
+                    id: task.uid,
+                });
+                return Ok((Some(Batch::IndexSwap { task }), current_batch));
+            };
         };
 
         let index_already_exists = self.index_mapper.exists(rtxn, index_name)?;
@@ -621,6 +697,10 @@ impl IndexScheduler {
                 .get_task(rtxn, task_id)
                 .and_then(|task| task.ok_or(Error::CorruptedTaskQueue))?;
 
+            if skip_if(&task) {
+                continue;
+            }
+
             if let Some(uuid) = task.content_uuid() {
                 let content_size = match self.queue.file_store.compute_size(uuid) {
                     Ok(content_size) => content_size,
@@ -651,19 +731,127 @@ impl IndexScheduler {
             autobatcher::autobatch(enqueued, index_already_exists, primary_key.as_deref())
         {
             current_batch.reason(autobatch_stop_reason.unwrap_or(stop_reason));
-            return Ok(self
+            let batch = self.create_next_batch_index(
-                .create_next_batch_index(
+                rtxn,
-                    rtxn,
+                index_name.to_string(),
-                    index_name.to_string(),
+                batchkind,
-                    batchkind,
+                &mut current_batch,
-                    &mut current_batch,
+                create_index,
-                    create_index,
+            )?;
-                )?
+            return Ok((batch, current_batch));
-                .map(|batch| (batch, current_batch)));
         }
 
         // If we found no tasks then we were notified for something that got autobatched
         // somehow and there is nothing to do.
-        Ok(None)
+        Ok((None, current_batch))
+    }
+
+    fn start_processing_network(
+        &self,
+        rtxn: &RoTxn,
+        mut task: Task,
+        enqueued: &RoaringBitmap,
+        mut current_batch: ProcessingBatch,
+    ) -> Result<Option<(Batch, ProcessingBatch)>> {
+        current_batch.processing(Some(&mut task));
+        current_batch.reason(BatchStopReason::NetworkTask { id: task.uid });
+
+        let change_version =
+            task.network.as_ref().map(|network| network.network_version()).unwrap_or_default();
+        let KindWithContent::NetworkTopologyChange(network_topology_change) = &task.kind else {
+            panic!("inconsistent kind with content")
+        };
+
+        match network_topology_change.state() {
+            NetworkTopologyState::WaitingForOlderTasks => {
+                let res =
+                    self.create_next_batch_unprioritized(rtxn, enqueued, current_batch, |task| {
+                        // in this limited mode of execution, we only want to run tasks:
+                        // 0. with an index
+                        // 1. with a version
+                        // 2. that version strictly lower than the network task version
+
+                        // 0. skip indexless tasks that are not index swap
+                        if task.index_uid().is_none() && task.kind.as_kind() != Kind::IndexSwap {
+                            return true;
+                        }
+
+                        // 1. skip tasks without version
+                        let Some(task_version) =
+                            task.network.as_ref().map(|network| network.network_version())
+                        else {
+                            return true;
+                        };
+
+                        // 2. skip tasks with a version equal or higher to the network task version
+                        task_version >= change_version
+                    });
+
+                let (batch, mut current_batch) = res?;
+
+                let batch = match batch {
+                    Some(batch) => {
+                        let inner_batch = Box::new(batch);
+                        let inner_reason = current_batch.reason.to_string();
+                        current_batch.reason(BatchStopReason::NetworkTaskOlderTasks {
+                            id: task.uid,
+                            inner_reason,
+                        });
+
+                        Batch::NetworkIndexBatch { network_task: task, inner_batch }
+                    }
+                    None => Batch::NetworkReady { task },
+                };
+
+                Ok(Some((batch, current_batch)))
+            }
+            NetworkTopologyState::ImportingDocuments => {
+                // if the import is done we need to go to the next state
+                if network_topology_change.is_import_finished() {
+                    return Ok(Some((Batch::NetworkReady { task }, current_batch)));
+                }
+
+                let res =
+                    self.create_next_batch_unprioritized(rtxn, enqueued, current_batch, |task| {
+                        // in this limited mode of execution, we only want to run tasks:
+                        // 0. with an index
+                        // 1. with a version
+                        // 2. that version equal to the network task version
+
+                        // 0. skip indexless tasks
+                        if task.index_uid().is_none() && task.kind.as_kind() != Kind::IndexSwap {
+                            return true;
+                        }
+
+                        // 1. skip tasks without version
+                        let Some(task_version) =
+                            task.network.as_ref().map(|network| network.network_version())
+                        else {
+                            return true;
+                        };
+
+                        // 2. skip tasks with a version different from the network task version
+                        task_version != change_version
+                    });
+
+                let (batch, mut current_batch) = res?;
+
+                let batch = batch.map(|batch| {
+                    let inner_batch = Box::new(batch);
+                    let inner_reason = current_batch.reason.to_string();
+                    current_batch.reason(BatchStopReason::NetworkTaskImportTasks {
+                        id: task.uid,
+                        inner_reason,
+                    });
+
+                    (Batch::NetworkIndexBatch { network_task: task, inner_batch }, current_batch)
+                });
+
+                Ok(batch)
+            }
+            NetworkTopologyState::ExportingDocuments | NetworkTopologyState::Finished => {
+                Ok(Some((Batch::NetworkReady { task }, current_batch)))
+            }
+        }
     }
 }

crates/index-scheduler/src/scheduler/enterprise_edition/mod.rs

@@ -0,0 +1,845 @@
+// Copyright © 2025 Meilisearch Some Rights Reserved
+// This file is part of Meilisearch Enterprise Edition (EE).
+// Use of this source code is governed by the Business Source License 1.1,
+// as found in the LICENSE-EE file or at <https://mariadb.com/bsl11>
+
+use std::time::Duration;
+
+use bumpalo::Bump;
+use roaring::RoaringBitmap;
+
+use meilisearch_types::milli::documents::PrimaryKey;
+use meilisearch_types::milli::progress::{EmbedderStats, Progress};
+use meilisearch_types::milli::update::new::indexer;
+use meilisearch_types::milli::update::new::indexer::current_edition::sharding::Shards;
+use meilisearch_types::milli::{self};
+use meilisearch_types::network::Remote;
+use meilisearch_types::tasks::network::{NetworkTopologyState, Origin};
+use meilisearch_types::tasks::{KindWithContent, Status, Task};
+
+use super::create_batch::Batch;
+use crate::scheduler::process_batch::ProcessBatchInfo;
+use crate::scheduler::process_export::{ExportContext, ExportOptions, TargetInstance};
+use crate::utils::ProcessingBatch;
+use crate::{Error, IndexScheduler, Result};
+
+impl IndexScheduler {
+    pub(super) fn process_network_index_batch(
+        &self,
+        mut network_task: Task,
+        inner_batch: Box<Batch>,
+        current_batch: &mut ProcessingBatch,
+        progress: Progress,
+    ) -> Result<(Vec<Task>, ProcessBatchInfo)> {
+        let KindWithContent::NetworkTopologyChange(network_topology_change) =
+            &mut network_task.kind
+        else {
+            tracing::error!("unexpected network kind for network task while processing batch");
+            return Err(Error::CorruptedTaskQueue);
+        };
+
+        let network = network_topology_change.network_for_state();
+
+        let (mut tasks, info) =
+            self.process_batch(*inner_batch, current_batch, progress, network)?;
+
+        for task in &tasks {
+            let Some(network) = task.network.as_ref() else {
+                continue;
+            };
+            let Some(import) = network.import_data() else {
+                continue;
+            };
+            if let Some(index_name) = import.index_name.as_deref() {
+                network_topology_change.process_remote_tasks(
+                    &import.remote_name,
+                    index_name,
+                    import.document_count,
+                );
+            }
+        }
+        network_task.details = Some(network_topology_change.to_details());
+
+        tasks.push(network_task);
+        Ok((tasks, info))
+    }
+
+    pub(super) fn process_network_ready(
+        &self,
+        mut task: Task,
+        progress: Progress,
+    ) -> Result<(Vec<Task>, ProcessBatchInfo)> {
+        let KindWithContent::NetworkTopologyChange(network_topology_change) = &mut task.kind else {
+            tracing::error!("network topology change task has the wrong kind with content");
+            return Err(Error::CorruptedTaskQueue);
+        };
+
+        let Some(task_network) = &task.network else {
+            tracing::error!("network topology change task has no network");
+            return Err(Error::CorruptedTaskQueue);
+        };
+
+        let origin;
+        let origin = match task_network.origin() {
+            Some(origin) => origin,
+            None => {
+                let myself = network_topology_change.in_name().expect("origin is not the leader");
+                origin = Origin {
+                    remote_name: myself.to_string(),
+                    task_uid: task.uid,
+                    network_version: task_network.network_version(),
+                };
+                &origin
+            }
+        };
+
+        let mut moved_documents = None;
+        if let (Some((remotes, out_name)), Some(new_shards)) =
+            (network_topology_change.export_to_process(), network_topology_change.new_shards())
+        {
+            moved_documents = Some(self.balance_documents(
+                remotes,
+                out_name,
+                new_shards,
+                origin,
+                &progress,
+                &self.scheduler.must_stop_processing,
+            )?);
+        }
+        if let Some(moved_documents) = moved_documents {
+            // we need the mut moved documents to avoid a lifetime error in the previous if let.
+            network_topology_change.set_moved(moved_documents);
+        }
+        network_topology_change.update_state();
+        if network_topology_change.state() == NetworkTopologyState::Finished {
+            task.status = Status::Succeeded;
+        }
+
+        task.details = Some(network_topology_change.to_details());
+        Ok((vec![task], Default::default()))
+    }
+
+    fn balance_documents<'a, I: Iterator<Item = (&'a str, &'a Remote)> + Clone>(
+        &self,
+        remotes: I,
+        out_name: &str,
+        new_shards: Shards,
+        network_change_origin: &Origin,
+        progress: &Progress,
+        must_stop_processing: &crate::scheduler::MustStopProcessing,
+    ) -> crate::Result<u64> {
+        // TECHDEBT: this spawns a `ureq` agent additionally to `reqwest`. We probably want to harmonize all of this.
+        let agent = ureq::AgentBuilder::new().timeout(Duration::from_secs(5)).build();
+
+        let mut indexer_alloc = Bump::new();
+
+        let scheduler_rtxn = self.env.read_txn()?;
+
+        let index_count = self.index_mapper.index_count(&scheduler_rtxn)?;
+
+        // when the instance is empty, we still need to tell that to remotes, as they cannot know of that fact and will be waiting for
+        // data
+        if index_count == 0 {
+            for (remote_name, remote) in remotes {
+                let target = TargetInstance {
+                    remote_name: Some(remote_name),
+                    base_url: &remote.url,
+                    api_key: remote.write_api_key.as_deref(),
+                };
+
+                let res = self.export_no_index(
+                    target,
+                    out_name,
+                    network_change_origin,
+                    &agent,
+                    must_stop_processing,
+                );
+
+                if let Err(err) = res {
+                    tracing::warn!("Could not signal not to wait documents to `{remote_name}` due to error: {err}");
+                }
+            }
+            return Ok(0);
+        }
+
+        let mut total_moved_documents = 0;
+
+        self.index_mapper.try_for_each_index::<(), ()>(
+            &scheduler_rtxn,
+            |index_uid, index| -> crate::Result<()> {
+                indexer_alloc.reset();
+                let err = |err| Error::from_milli(err, Some(index_uid.to_string()));
+                let index_rtxn = index.read_txn()?;
+                let all_docids = index.external_documents_ids();
+                let mut documents_to_move_to =
+                    hashbrown::HashMap::<String, RoaringBitmap>::new();
+                let mut documents_to_delete = RoaringBitmap::new();
+
+                for res in all_docids.iter(&index_rtxn)? {
+                    let (external_docid, docid) = res?;
+                    match new_shards.processing_shard(external_docid) {
+                        Some(shard) if shard.is_own => continue,
+                        Some(shard) => {
+                            documents_to_move_to.entry_ref(&shard.name).or_default().insert(docid);
+                        }
+                        None => {
+                            documents_to_delete.insert(docid);
+                        }
+                    }
+                }
+
+                let fields_ids_map = index.fields_ids_map(&index_rtxn)?;
+
+                for (remote_name, remote) in remotes.clone() {
+                    let documents_to_move =
+                        documents_to_move_to.remove(remote_name).unwrap_or_default();
+
+                    let target = TargetInstance {
+                        remote_name: Some(remote_name),
+                        base_url: &remote.url,
+                        api_key: remote.write_api_key.as_deref(),
+                    };
+                    let options = ExportOptions {
+                        index_uid,
+                        payload_size: None,
+                        override_settings: false,
+                        export_mode: super::process_export::ExportMode::NetworkBalancing {
+                            index_count,
+                            export_old_remote_name: out_name,
+                            network_change_origin,
+                        },
+                    };
+                    let ctx = ExportContext {
+                        index,
+                        index_rtxn: &index_rtxn,
+                        universe: &documents_to_move,
+                        progress,
+                        agent: &agent,
+                        must_stop_processing,
+                    };
+
+                    let res = self.export_one_index(target, options, ctx);
+
+                    match res {
+                        Ok(_) =>{ documents_to_delete |= documents_to_move;}
+                        Err(err) => {
+                            tracing::warn!("Could not export documents to `{remote_name}` due to error: {err}\n  - Note: Documents will be kept");
+                        }
+                    }
+
+
+                }
+
+                if documents_to_delete.is_empty() {
+                    return Ok(());
+                }
+
+                total_moved_documents += documents_to_delete.len();
+
+                self.delete_documents_from_index(progress, must_stop_processing, &indexer_alloc, index_uid, index, &err, index_rtxn, documents_to_delete, fields_ids_map)
+            },
+        )?;
+
+        Ok(total_moved_documents)
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    fn delete_documents_from_index(
+        &self,
+        progress: &Progress,
+        must_stop_processing: &super::MustStopProcessing,
+        indexer_alloc: &Bump,
+        index_uid: &str,
+        index: &milli::Index,
+        err: &impl Fn(milli::Error) -> Error,
+        index_rtxn: milli::heed::RoTxn<'_, milli::heed::WithoutTls>,
+        documents_to_delete: RoaringBitmap,
+        fields_ids_map: milli::FieldsIdsMap,
+    ) -> std::result::Result<(), Error> {
+        let mut new_fields_ids_map = fields_ids_map.clone();
+
+        // candidates not empty => index not empty => a primary key is set
+        let primary_key = index.primary_key(&index_rtxn)?.unwrap();
+
+        let primary_key = PrimaryKey::new_or_insert(primary_key, &mut new_fields_ids_map)
+            .map_err(milli::Error::from)
+            .map_err(err)?;
+
+        let mut index_wtxn = index.write_txn()?;
+
+        let mut indexer = indexer::DocumentDeletion::new();
+        indexer.delete_documents_by_docids(documents_to_delete);
+        let document_changes = indexer.into_changes(indexer_alloc, primary_key);
+        let embedders = index
+            .embedding_configs()
+            .embedding_configs(&index_wtxn)
+            .map_err(milli::Error::from)
+            .map_err(err)?;
+        let embedders = self.embedders(index_uid.to_string(), embedders)?;
+        let indexer_config = self.index_mapper.indexer_config();
+        let pool = &indexer_config.thread_pool;
+
+        indexer::index(
+            &mut index_wtxn,
+            index,
+            pool,
+            indexer_config.grenad_parameters(),
+            &fields_ids_map,
+            new_fields_ids_map,
+            None, // document deletion never changes primary key
+            &document_changes,
+            embedders,
+            &|| must_stop_processing.get(),
+            progress,
+            &EmbedderStats::default(),
+        )
+        .map_err(err)?;
+
+        // update stats
+        let mut mapper_wtxn = self.env.write_txn()?;
+        let stats = crate::index_mapper::IndexStats::new(index, &index_wtxn).map_err(err)?;
+        self.index_mapper.store_stats_of(&mut mapper_wtxn, index_uid, &stats)?;
+
+        index_wtxn.commit()?;
+        // update stats after committing changes to index
+        mapper_wtxn.commit()?;
+
+        Ok(())
+    }
+
+    #[cfg(unix)]
+    async fn assume_role_with_web_identity(
+        role_arn: &str,
+        web_identity_token_file: &std::path::Path,
+    ) -> anyhow::Result<StsCredentials> {
+        use std::env::VarError;
+
+        let token = tokio::fs::read_to_string(web_identity_token_file)
+            .await
+            .map_err(|e| anyhow::anyhow!("Failed to read web identity token file: {e}"))?;
+
+        let duration: u32 =
+            match std::env::var("MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_DURATION_SECONDS") {
+                Ok(s) => s.parse()?,
+                Err(VarError::NotPresent) => 3600,
+                Err(VarError::NotUnicode(e)) => {
+                    anyhow::bail!("Invalid duration: {e:?}")
+                }
+            };
+
+        let form_data = [
+            ("Action", "AssumeRoleWithWebIdentity"),
+            ("Version", "2011-06-15"),
+            ("RoleArn", role_arn),
+            ("RoleSessionName", "meilisearch-snapshot-session"),
+            ("WebIdentityToken", &token),
+            ("DurationSeconds", &duration.to_string()),
+        ];
+
+        let client = reqwest::Client::new();
+        let response = client
+            .post("https://sts.amazonaws.com/")
+            .header(reqwest::header::ACCEPT, "application/json")
+            .header(reqwest::header::CONTENT_TYPE, "application/x-www-form-urlencoded")
+            .form(&form_data)
+            .send()
+            .await
+            .map_err(|e| anyhow::anyhow!("Failed to send STS request: {e}"))?;
+
+        let status = response.status();
+        let body = response
+            .text()
+            .await
+            .map_err(|e| anyhow::anyhow!("Failed to read STS response body: {e}"))?;
+
+        if !status.is_success() {
+            return Err(anyhow::anyhow!("STS request failed with status {status}: {body}"));
+        }
+
+        let sts_response: StsResponse = serde_json::from_str(&body)
+            .map_err(|e| anyhow::anyhow!("Failed to deserialize STS response: {e}"))?;
+
+        Ok(sts_response.response.result.credentials)
+    }
+
+    #[cfg(unix)]
+    async fn extract_credentials_from_options(
+        s3_access_key: Option<String>,
+        s3_secret_key: Option<String>,
+        s3_role_arn: Option<String>,
+        s3_web_identity_token_file: Option<std::path::PathBuf>,
+    ) -> anyhow::Result<(String, String, Option<String>)> {
+        let static_credentials = s3_access_key.zip(s3_secret_key);
+        let web_identity = s3_role_arn.zip(s3_web_identity_token_file);
+        match (static_credentials, web_identity) {
+            (Some((access_key, secret_key)), None) => Ok((access_key, secret_key, None)),
+            (None, Some((role_arn, token_file))) => {
+                let StsCredentials { access_key_id, secret_access_key, session_token } =
+                    Self::assume_role_with_web_identity(&role_arn, &token_file).await?;
+                Ok((access_key_id, secret_access_key, Some(session_token)))
+            }
+            (_, _) => anyhow::bail!("Clap must pass valid auth parameters"),
+        }
+    }
+
+    #[cfg(unix)]
+    pub(super) async fn process_snapshot_to_s3(
+        &self,
+        progress: Progress,
+        opts: meilisearch_types::milli::update::S3SnapshotOptions,
+        mut tasks: Vec<Task>,
+    ) -> Result<Vec<Task>> {
+        use meilisearch_types::milli::update::S3SnapshotOptions;
+        use std::ffi::OsStr;
+
+        let S3SnapshotOptions {
+            s3_bucket_url,
+            s3_bucket_region,
+            s3_bucket_name,
+            s3_snapshot_prefix,
+            s3_access_key,
+            s3_secret_key,
+            s3_role_arn,
+            s3_web_identity_token_file,
+            s3_max_in_flight_parts,
+            s3_compression_level: level,
+            s3_signature_duration,
+            s3_multipart_part_size,
+        } = opts;
+
+        let must_stop_processing = self.scheduler.must_stop_processing.clone();
+        let retry_backoff = backoff::ExponentialBackoff::default();
+        let db_name = {
+            let mut base_path = self.env.path().to_owned();
+            base_path.pop();
+            base_path.file_name().and_then(OsStr::to_str).unwrap_or("data.ms").to_string()
+        };
+
+        let (reader, writer) = std::io::pipe()?;
+        let uploader_task = tokio::spawn(async move {
+            let (s3_access_key, s3_secret_key, s3_token) = Self::extract_credentials_from_options(
+                s3_access_key,
+                s3_secret_key,
+                s3_role_arn,
+                s3_web_identity_token_file,
+            )
+            .await?;
+
+            multipart_stream_to_s3(
+                s3_bucket_url,
+                s3_bucket_region,
+                s3_bucket_name,
+                s3_snapshot_prefix,
+                s3_access_key,
+                s3_secret_key,
+                s3_token,
+                s3_max_in_flight_parts,
+                s3_signature_duration,
+                s3_multipart_part_size,
+                must_stop_processing,
+                retry_backoff,
+                db_name,
+                reader,
+            )
+            .await
+        });
+
+        let index_scheduler = IndexScheduler::private_clone(self);
+        let builder_task = tokio::task::spawn_blocking(move || {
+            stream_tarball_into_pipe(progress, level, writer, index_scheduler)
+        });
+
+        let (uploader_result, builder_result) = tokio::join!(uploader_task, builder_task);
+
+        // Check uploader result first to early return on task abortion.
+        // safety: JoinHandle can return an error if the task was aborted, cancelled, or panicked.
+        uploader_result.unwrap()?;
+        builder_result.unwrap()?;
+
+        for task in &mut tasks {
+            task.status = Status::Succeeded;
+        }
+
+        Ok(tasks)
+    }
+}
+
+#[cfg(unix)]
+#[derive(Debug, Clone, serde::Deserialize)]
+struct StsCredentials {
+    #[serde(rename = "AccessKeyId")]
+    access_key_id: String,
+    #[serde(rename = "SecretAccessKey")]
+    secret_access_key: String,
+    #[serde(rename = "SessionToken")]
+    session_token: String,
+}
+
+#[cfg(unix)]
+#[derive(Debug, serde::Deserialize)]
+struct AssumeRoleWithWebIdentityResult {
+    #[serde(rename = "Credentials")]
+    credentials: StsCredentials,
+}
+
+#[cfg(unix)]
+#[derive(Debug, serde::Deserialize)]
+struct AssumeRoleWithWebIdentityResponse {
+    #[serde(rename = "AssumeRoleWithWebIdentityResult")]
+    result: AssumeRoleWithWebIdentityResult,
+}
+
+#[cfg(unix)]
+#[derive(Debug, serde::Deserialize)]
+struct StsResponse {
+    #[serde(rename = "AssumeRoleWithWebIdentityResponse")]
+    response: AssumeRoleWithWebIdentityResponse,
+}
+
+/// Streams a tarball of the database content into a pipe.
+#[cfg(unix)]
+fn stream_tarball_into_pipe(
+    progress: Progress,
+    level: u32,
+    writer: std::io::PipeWriter,
+    index_scheduler: IndexScheduler,
+) -> std::result::Result<(), Error> {
+    use std::io::Write as _;
+    use std::path::Path;
+    use std::sync::atomic::Ordering;
+
+    use meilisearch_types::milli::progress::VariableNameStep;
+    use meilisearch_types::VERSION_FILE_NAME;
+
+    use crate::processing::{AtomicUpdateFileStep, SnapshotCreationProgress};
+    use crate::scheduler::process_snapshot_creation::UPDATE_FILES_DIR_NAME;
+
+    let writer = flate2::write::GzEncoder::new(writer, flate2::Compression::new(level));
+    let mut tarball = tar::Builder::new(writer);
+
+    // 1. Snapshot the version file
+    tarball
+        .append_path_with_name(&index_scheduler.scheduler.version_file_path, VERSION_FILE_NAME)?;
+
+    // 2. Snapshot the index scheduler LMDB env
+    progress.update_progress(SnapshotCreationProgress::SnapshotTheIndexScheduler);
+    let tasks_env_file = index_scheduler.env.try_clone_inner_file()?;
+    let path = Path::new("tasks").join("data.mdb");
+    append_file_to_tarball(&mut tarball, path, tasks_env_file)?;
+
+    // 2.3 Create a read transaction on the index-scheduler
+    let rtxn = index_scheduler.env.read_txn()?;
+
+    // 2.4 Create the update files directory
+    //     And only copy the update files of the enqueued tasks
+    progress.update_progress(SnapshotCreationProgress::SnapshotTheUpdateFiles);
+    let enqueued = index_scheduler.queue.tasks.get_status(&rtxn, Status::Enqueued)?;
+    let (atomic, update_file_progress) = AtomicUpdateFileStep::new(enqueued.len() as u32);
+    progress.update_progress(update_file_progress);
+
+    // We create the update_files directory so that it
+    // always exists even if there are no update files
+    let update_files_dir = Path::new(UPDATE_FILES_DIR_NAME);
+    let src_update_files_dir = {
+        let mut path = index_scheduler.env.path().to_path_buf();
+        path.pop();
+        path.join(UPDATE_FILES_DIR_NAME)
+    };
+    tarball.append_dir(update_files_dir, src_update_files_dir)?;
+
+    for task_id in enqueued {
+        let task = index_scheduler
+            .queue
+            .tasks
+            .get_task(&rtxn, task_id)?
+            .ok_or(Error::CorruptedTaskQueue)?;
+        if let Some(content_uuid) = task.content_uuid() {
+            use std::fs::File;
+
+            let src = index_scheduler.queue.file_store.update_path(content_uuid);
+            let mut update_file = File::open(src)?;
+            let path = update_files_dir.join(content_uuid.to_string());
+            tarball.append_file(path, &mut update_file)?;
+        }
+        atomic.fetch_add(1, Ordering::Relaxed);
+    }
+
+    // 3. Snapshot every indexes
+    progress.update_progress(SnapshotCreationProgress::SnapshotTheIndexes);
+    let index_mapping = index_scheduler.index_mapper.index_mapping;
+    let nb_indexes = index_mapping.len(&rtxn)? as u32;
+    let indexes_dir = Path::new("indexes");
+    let indexes_references: Vec<_> = index_scheduler
+        .index_mapper
+        .index_mapping
+        .iter(&rtxn)?
+        .map(|res| res.map_err(Error::from).map(|(name, uuid)| (name.to_string(), uuid)))
+        .collect::<Result<_, Error>>()?;
+
+    // It's prettier to use a for loop instead of the IndexMapper::try_for_each_index
+    // method, especially when we need to access the UUID, local path and index number.
+    for (i, (name, uuid)) in indexes_references.into_iter().enumerate() {
+        progress.update_progress(VariableNameStep::<SnapshotCreationProgress>::new(
+            &name, i as u32, nb_indexes,
+        ));
+        let path = indexes_dir.join(uuid.to_string()).join("data.mdb");
+        let index = index_scheduler.index_mapper.index(&rtxn, &name)?;
+        let index_file = index.try_clone_inner_file()?;
+        tracing::trace!("Appending index file for {name} in {}", path.display());
+        append_file_to_tarball(&mut tarball, path, index_file)?;
+    }
+
+    drop(rtxn);
+
+    // 4. Snapshot the auth LMDB env
+    progress.update_progress(SnapshotCreationProgress::SnapshotTheApiKeys);
+    let auth_env_file = index_scheduler.scheduler.auth_env.try_clone_inner_file()?;
+    let path = Path::new("auth").join("data.mdb");
+    append_file_to_tarball(&mut tarball, path, auth_env_file)?;
+
+    let mut gzencoder = tarball.into_inner()?;
+    gzencoder.flush()?;
+    gzencoder.try_finish()?;
+    let mut writer = gzencoder.finish()?;
+    writer.flush()?;
+
+    Result::<_, Error>::Ok(())
+}
+
+#[cfg(unix)]
+fn append_file_to_tarball<W, P>(
+    tarball: &mut tar::Builder<W>,
+    path: P,
+    mut auth_env_file: std::fs::File,
+) -> Result<(), Error>
+where
+    W: std::io::Write,
+    P: AsRef<std::path::Path>,
+{
+    use std::io::{Seek as _, SeekFrom};
+
+    // Note: A previous snapshot operation may have left the cursor
+    //       at the end of the file so we need to seek to the start.
+    auth_env_file.seek(SeekFrom::Start(0))?;
+    tarball.append_file(path, &mut auth_env_file)?;
+    Ok(())
+}
+
+/// Streams the content read from the given reader to S3.
+#[allow(clippy::too_many_arguments)]
+#[cfg(unix)]
+async fn multipart_stream_to_s3(
+    s3_bucket_url: String,
+    s3_bucket_region: String,
+    s3_bucket_name: String,
+    s3_snapshot_prefix: String,
+    s3_access_key: String,
+    s3_secret_key: String,
+    s3_token: Option<String>,
+    s3_max_in_flight_parts: std::num::NonZero<usize>,
+    s3_signature_duration: std::time::Duration,
+    s3_multipart_part_size: u64,
+    must_stop_processing: super::MustStopProcessing,
+    retry_backoff: backoff::exponential::ExponentialBackoff<backoff::SystemClock>,
+    db_name: String,
+    reader: std::io::PipeReader,
+) -> Result<(), Error> {
+    use std::collections::VecDeque;
+    use std::io;
+    use std::os::fd::OwnedFd;
+    use std::path::PathBuf;
+
+    use bytes::{Bytes, BytesMut};
+    use reqwest::{Client, Response};
+    use rusty_s3::actions::CreateMultipartUpload;
+    use rusty_s3::{Bucket, BucketError, Credentials, S3Action as _, UrlStyle};
+    use tokio::task::JoinHandle;
+
+    let reader = OwnedFd::from(reader);
+    let reader = tokio::net::unix::pipe::Receiver::from_owned_fd(reader)?;
+    let s3_snapshot_prefix = PathBuf::from(s3_snapshot_prefix);
+    let url =
+        s3_bucket_url.parse().map_err(BucketError::ParseError).map_err(Error::S3BucketError)?;
+    let bucket = Bucket::new(url, UrlStyle::Path, s3_bucket_name, s3_bucket_region)
+        .map_err(Error::S3BucketError)?;
+    let credential = match s3_token {
+        Some(token) => Credentials::new_with_token(s3_access_key, s3_secret_key, token),
+        None => Credentials::new(s3_access_key, s3_secret_key),
+    };
+
+    // Note for the future (rust 1.91+): use with_added_extension, it's prettier
+    let object_path = s3_snapshot_prefix.join(format!("{db_name}.snapshot"));
+    // Note: It doesn't work on Windows and if a port to this platform is needed,
+    //       use the slash-path crate or similar to get the correct path separator.
+    let object = object_path.display().to_string();
+
+    let action = bucket.create_multipart_upload(Some(&credential), &object);
+    let url = action.sign(s3_signature_duration);
+
+    let client = Client::new();
+    let resp = client.post(url).send().await.map_err(Error::S3HttpError)?;
+    let status = resp.status();
+
+    let body = match resp.error_for_status_ref() {
+        Ok(_) => resp.text().await.map_err(Error::S3HttpError)?,
+        Err(_) => {
+            return Err(Error::S3Error { status, body: resp.text().await.unwrap_or_default() })
+        }
+    };
+
+    let multipart =
+        CreateMultipartUpload::parse_response(&body).map_err(|e| Error::S3XmlError(Box::new(e)))?;
+    tracing::debug!("Starting the upload of the snapshot to {object}");
+
+    // We use this bumpalo for etags strings.
+    let bump = bumpalo::Bump::new();
+    let mut etags = Vec::<&str>::new();
+    let mut in_flight = VecDeque::<(JoinHandle<reqwest::Result<Response>>, Bytes)>::with_capacity(
+        s3_max_in_flight_parts.get(),
+    );
+
+    // Part numbers start at 1 and cannot be larger than 10k
+    for part_number in 1u16.. {
+        if must_stop_processing.get() {
+            return Err(Error::AbortedTask);
+        }
+
+        let part_upload =
+            bucket.upload_part(Some(&credential), &object, part_number, multipart.upload_id());
+        let url = part_upload.sign(s3_signature_duration);
+
+        // Wait for a buffer to be ready if there are in-flight parts that landed
+        let mut buffer = if in_flight.len() >= s3_max_in_flight_parts.get() {
+            let (handle, buffer) = in_flight.pop_front().expect("At least one in flight request");
+            let resp = join_and_map_error(handle).await?;
+            extract_and_append_etag(&bump, &mut etags, resp.headers())?;
+
+            let mut buffer = match buffer.try_into_mut() {
+                Ok(buffer) => buffer,
+                Err(_) => unreachable!("All bytes references were consumed in the task"),
+            };
+            buffer.clear();
+            buffer
+        } else {
+            BytesMut::with_capacity(s3_multipart_part_size as usize)
+        };
+
+        // If we successfully read enough bytes,
+        // we can continue and send the buffer/part
+        while buffer.len() < (s3_multipart_part_size as usize / 2) {
+            // Wait for the pipe to be readable
+
+            reader.readable().await?;
+
+            match reader.try_read_buf(&mut buffer) {
+                Ok(0) => break,
+                // We read some bytes but maybe not enough
+                Ok(_) => continue,
+                // The readiness event is a false positive.
+                Err(ref e) if e.kind() == io::ErrorKind::WouldBlock => continue,
+                Err(e) => return Err(e.into()),
+            }
+        }
+
+        if buffer.is_empty() {
+            // Break the loop if the buffer is
+            // empty after we tried to read bytes
+            break;
+        }
+
+        let body = buffer.freeze();
+        tracing::trace!("Sending part {part_number}");
+        let task = tokio::spawn({
+            let client = client.clone();
+            let body = body.clone();
+            backoff::future::retry(retry_backoff.clone(), move || {
+                let client = client.clone();
+                let url = url.clone();
+                let body = body.clone();
+                async move {
+                    match client.put(url).body(body).send().await {
+                        Ok(resp) if resp.status().is_client_error() => {
+                            resp.error_for_status().map_err(backoff::Error::Permanent)
+                        }
+                        Ok(resp) => Ok(resp),
+                        Err(e) => Err(backoff::Error::transient(e)),
+                    }
+                }
+            })
+        });
+        in_flight.push_back((task, body));
+    }
+
+    for (handle, _buffer) in in_flight {
+        let resp = join_and_map_error(handle).await?;
+        extract_and_append_etag(&bump, &mut etags, resp.headers())?;
+    }
+
+    tracing::debug!("Finalizing the multipart upload");
+
+    let action = bucket.complete_multipart_upload(
+        Some(&credential),
+        &object,
+        multipart.upload_id(),
+        etags.iter().map(AsRef::as_ref),
+    );
+    let url = action.sign(s3_signature_duration);
+    let body = action.body();
+    let resp = backoff::future::retry(retry_backoff, move || {
+        let client = client.clone();
+        let url = url.clone();
+        let body = body.clone();
+        async move {
+            match client.post(url).body(body).send().await {
+                Ok(resp) if resp.status().is_client_error() => {
+                    Err(backoff::Error::Permanent(Error::S3Error {
+                        status: resp.status(),
+                        body: resp.text().await.unwrap_or_default(),
+                    }))
+                }
+                Ok(resp) => Ok(resp),
+                Err(e) => Err(backoff::Error::transient(Error::S3HttpError(e))),
+            }
+        }
+    })
+    .await?;
+
+    let status = resp.status();
+    let body = resp.text().await.map_err(|e| Error::S3Error { status, body: e.to_string() })?;
+    if status.is_success() {
+        Ok(())
+    } else {
+        Err(Error::S3Error { status, body })
+    }
+}
+
+#[cfg(unix)]
+async fn join_and_map_error(
+    join_handle: tokio::task::JoinHandle<Result<reqwest::Response, reqwest::Error>>,
+) -> Result<reqwest::Response> {
+    // safety: Panic happens if the task (JoinHandle) was aborted, cancelled, or panicked
+    let request = join_handle.await.unwrap();
+    let resp = request.map_err(Error::S3HttpError)?;
+    match resp.error_for_status_ref() {
+        Ok(_) => Ok(resp),
+        Err(_) => Err(Error::S3Error {
+            status: resp.status(),
+            body: resp.text().await.unwrap_or_default(),
+        }),
+    }
+}
+
+#[cfg(unix)]
+fn extract_and_append_etag<'b>(
+    bump: &'b bumpalo::Bump,
+    etags: &mut Vec<&'b str>,
+    headers: &reqwest::header::HeaderMap,
+) -> Result<()> {
+    use reqwest::header::ETAG;
+
+    let etag = headers.get(ETAG).ok_or_else(|| Error::S3XmlError("Missing ETag header".into()))?;
+    let etag = etag.to_str().map_err(|e| Error::S3XmlError(Box::new(e)))?;
+    etags.push(bump.alloc_str(etag));
+
+    Ok(())
+}

crates/index-scheduler/src/scheduler/mod.rs

@@ -1,7 +1,12 @@
 mod autobatcher;
 #[cfg(test)]
 mod autobatcher_test;
+#[cfg(not(feature = "enterprise"))]
+mod community_edition;
 mod create_batch;
+#[cfg(feature = "enterprise")]
+mod enterprise_edition;
+
 mod process_batch;
 mod process_dump_creation;
 mod process_export;
@@ -21,7 +26,6 @@ use std::path::PathBuf;
 use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
 use std::sync::Arc;
 
-use convert_case::{Case, Casing as _};
 use meilisearch_types::error::ResponseError;
 use meilisearch_types::heed::{Env, WithoutTls};
 use meilisearch_types::milli;
@@ -133,6 +137,7 @@ impl Scheduler {
             max_number_of_tasks: _,
             max_number_of_batched_tasks,
             batched_tasks_size_limit,
+            export_default_payload_size_bytes: _,
             instance_features: _,
             auto_upgrade: _,
             embedding_cache_cap,
@@ -178,6 +183,8 @@ impl IndexScheduler {
             self.breakpoint(crate::test_utils::Breakpoint::Start);
         }
 
+        let previous_processing_batch = self.processing_tasks.write().unwrap().stop_processing();
+
         if self.cleanup_enabled {
             let mut wtxn = self.env.write_txn()?;
             self.queue.cleanup_task_queue(&mut wtxn)?;
@@ -185,11 +192,16 @@ impl IndexScheduler {
         }
 
         let rtxn = self.env.read_txn().map_err(Error::HeedTransaction)?;
-        let (batch, mut processing_batch) =
+        let (batch, mut processing_batch) = match self
-            match self.create_next_batch(&rtxn).map_err(|e| Error::CreateBatch(Box::new(e)))? {
+            .create_next_batch(&rtxn, &previous_processing_batch.processing)
-                Some(batch) => batch,
+            .map_err(|e| Error::CreateBatch(Box::new(e)))?
-                None => return Ok(TickOutcome::WaitForSignal),
+        {
-            };
+            Some(batch) => batch,
+            None => {
+                *self.processing_tasks.write().unwrap() = previous_processing_batch;
+                return Ok(TickOutcome::WaitForSignal);
+            }
+        };
         let index_uid = batch.index_uid().map(ToOwned::to_owned);
         drop(rtxn);
 
@@ -219,7 +231,12 @@ impl IndexScheduler {
                 let handle = std::thread::Builder::new()
                     .name(String::from("batch-operation"))
                     .spawn_scoped(s, move || {
-                        cloned_index_scheduler.process_batch(batch, processing_batch, p)
+                        cloned_index_scheduler.process_batch(
+                            batch,
+                            processing_batch,
+                            p,
+                            &self.network(),
+                        )
                     })
                     .unwrap();
 
@@ -260,7 +277,14 @@ impl IndexScheduler {
         self.maybe_fail(crate::test_utils::FailureLocation::AcquiringWtxn)?;
 
         progress.update_progress(BatchProgress::WritingTasksToDisk);
+
         processing_batch.finished();
+        // whether the batch made progress.
+        // a batch make progress if it failed or if it contains at least one fully processed (or cancelled) task.
+        //
+        // if a batch did not make progress, it means that all of its tasks are waiting on the scheduler to make progress,
+        // and so we must wait for new tasks. Such a batch is not persisted to DB, and is resumed on the next tick.
+        let mut batch_made_progress = false;
         let mut stop_scheduler_forever = false;
         let mut wtxn = self.env.write_txn().map_err(Error::HeedTransaction)?;
         let mut canceled = RoaringBitmap::new();
@@ -281,7 +305,11 @@ impl IndexScheduler {
                 #[allow(unused_variables)]
                 for (i, mut task) in tasks.into_iter().enumerate() {
                     task_progress.fetch_add(1, Ordering::Relaxed);
-                    processing_batch.update(&mut task);
+                    processing_batch.update_from_task(&task);
+                    if !matches!(task.status, Status::Processing | Status::Enqueued) {
+                        batch_made_progress = true;
+                        processing_batch.finish_task(&mut task);
+                    }
                     if task.status == Status::Canceled {
                         canceled.insert(task.uid);
                         canceled_by = task.canceled_by;
@@ -348,6 +376,9 @@ impl IndexScheduler {
             }
             // In case of a failure we must get back and patch all the tasks with the error.
             Err(err) => {
+                // always persist failed batches
+                batch_made_progress = true;
+
                 #[cfg(test)]
                 self.breakpoint(crate::test_utils::Breakpoint::ProcessBatchFailed);
                 let (task_progress, task_progress_obj) = AtomicTaskStep::new(ids.len() as u32);
@@ -371,7 +402,10 @@ impl IndexScheduler {
                     task.status = Status::Failed;
                     task.error = Some(error.clone());
                     task.details = task.details.map(|d| d.to_failed());
-                    processing_batch.update(&mut task);
+                    processing_batch.update_from_task(&task);
+                    if !matches!(task.status, Status::Processing | Status::Enqueued) {
+                        processing_batch.finish_task(&mut task);
+                    }
 
                     #[cfg(test)]
                     self.maybe_fail(
@@ -394,44 +428,12 @@ impl IndexScheduler {
         let ProcessBatchInfo { congestion, pre_commit_dabases_sizes, post_commit_dabases_sizes } =
             process_batch_info;
 
-        processing_batch.stats.progress_trace =
+        processing_batch.write_stats(
-            progress.accumulated_durations().into_iter().map(|(k, v)| (k, v.into())).collect();
+            &progress,
-        processing_batch.stats.write_channel_congestion = congestion.map(|congestion| {
+            congestion,
-            let mut congestion_info = serde_json::Map::new();
+            pre_commit_dabases_sizes,
-            congestion_info.insert("attempts".into(), congestion.attempts.into());
+            post_commit_dabases_sizes,
-            congestion_info.insert("blocking_attempts".into(), congestion.blocking_attempts.into());
+        );
-            congestion_info.insert("blocking_ratio".into(), congestion.congestion_ratio().into());
-            congestion_info
-        });
-        processing_batch.stats.internal_database_sizes = pre_commit_dabases_sizes
-            .iter()
-            .flat_map(|(dbname, pre_size)| {
-                post_commit_dabases_sizes
-                    .get(dbname)
-                    .map(|post_size| {
-                        use std::cmp::Ordering::{Equal, Greater, Less};
-
-                        use byte_unit::Byte;
-                        use byte_unit::UnitType::Binary;
-
-                        let post = Byte::from_u64(*post_size as u64).get_appropriate_unit(Binary);
-                        let diff_size = post_size.abs_diff(*pre_size) as u64;
-                        let diff = Byte::from_u64(diff_size).get_appropriate_unit(Binary);
-                        let sign = match post_size.cmp(pre_size) {
-                            Equal => return None,
-                            Greater => "+",
-                            Less => "-",
-                        };
-
-                        Some((
-                            dbname.to_case(Case::Camel),
-                            format!("{post:#.2} ({sign}{diff:#.2})").into(),
-                        ))
-                    })
-                    .into_iter()
-                    .flatten()
-            })
-            .collect();
 
         if let Some(congestion) = congestion {
             tracing::debug!(
@@ -444,46 +446,49 @@ impl IndexScheduler {
 
         tracing::debug!("call trace: {:?}", progress.accumulated_durations());
 
-        self.queue.write_batch(&mut wtxn, processing_batch, &ids)?;
+        if batch_made_progress {
+            self.queue.write_batch(&mut wtxn, processing_batch, &ids)?;
+        }
 
         #[cfg(test)]
         self.maybe_fail(crate::test_utils::FailureLocation::CommittingWtxn)?;
 
         wtxn.commit().map_err(Error::HeedTransaction)?;
 
-        // We should stop processing AFTER everything is processed and written to disk otherwise, a batch (which only lives in RAM) may appear in the processing task
+        if batch_made_progress {
-        // and then become « not found » for some time until the commit everything is written and the final commit is made.
+            // We should stop processing AFTER everything is processed and written to disk otherwise, a batch (which only lives in RAM) may appear in the processing task
-        self.processing_tasks.write().unwrap().stop_processing();
+            // and then become « not found » for some time until the commit everything is written and the final commit is made.
+            self.processing_tasks.write().unwrap().stop_processing();
 
-        // Once the tasks are committed, we should delete all the update files associated ASAP to avoid leaking files in case of a restart
+            // Once the tasks are committed, we should delete all the update files associated ASAP to avoid leaking files in case of a restart
-        tracing::debug!("Deleting the update files");
+            tracing::debug!("Deleting the update files");
 
-        //We take one read transaction **per thread**. Then, every thread is going to pull out new IDs from the roaring bitmap with the help of an atomic shared index into the bitmap
+            //We take one read transaction **per thread**. Then, every thread is going to pull out new IDs from the roaring bitmap with the help of an atomic shared index into the bitmap
-        let idx = AtomicU32::new(0);
+            let idx = AtomicU32::new(0);
-        (0..current_num_threads()).into_par_iter().try_for_each(|_| -> Result<()> {
+            (0..current_num_threads()).into_par_iter().try_for_each(|_| -> Result<()> {
-            let rtxn = self.read_txn()?;
+                let rtxn = self.read_txn()?;
-            while let Some(id) = ids.select(idx.fetch_add(1, Ordering::Relaxed)) {
+                while let Some(id) = ids.select(idx.fetch_add(1, Ordering::Relaxed)) {
-                let task = self
+                    let task = self
-                    .queue
+                        .queue
-                    .tasks
+                        .tasks
-                    .get_task(&rtxn, id)
+                        .get_task(&rtxn, id)
-                    .map_err(|e| Error::UnrecoverableError(Box::new(e)))?
+                        .map_err(|e| Error::UnrecoverableError(Box::new(e)))?
-                    .ok_or(Error::CorruptedTaskQueue)?;
+                        .ok_or(Error::CorruptedTaskQueue)?;
-                if let Err(e) = self.queue.delete_persisted_task_data(&task) {
+                    if let Err(e) = self.queue.delete_persisted_task_data(&task) {
-                    tracing::error!(
+                        tracing::error!(
                         "Failure to delete the content files associated with task {}. Error: {e}",
                         task.uid
                     );
+                    }
                 }
-            }
+                Ok(())
-            Ok(())
+            })?;
-        })?;
 
-        self.notify_webhooks(ids);
+            self.notify_webhooks(ids);
+        }
 
         #[cfg(test)]
         self.breakpoint(crate::test_utils::Breakpoint::AfterProcessing);
-
         if stop_scheduler_forever {
             Ok(TickOutcome::StopProcessingForever)
         } else {

crates/index-scheduler/src/scheduler/process_batch.rs

@@ -10,6 +10,7 @@ use meilisearch_types::heed::{RoTxn, RwTxn};
 use meilisearch_types::milli::heed::CompactionOption;
 use meilisearch_types::milli::progress::{Progress, VariableNameStep};
 use meilisearch_types::milli::{self, ChannelCongestion};
+use meilisearch_types::network::Network;
 use meilisearch_types::tasks::{Details, IndexSwap, Kind, KindWithContent, Status, Task};
 use meilisearch_types::versioning::{VERSION_MAJOR, VERSION_MINOR, VERSION_PATCH};
 use milli::update::Settings as MilliSettings;
@@ -55,6 +56,7 @@ impl IndexScheduler {
         batch: Batch,
         current_batch: &mut ProcessingBatch,
         progress: Progress,
+        network: &Network,
     ) -> Result<(Vec<Task>, ProcessBatchInfo)> {
         #[cfg(test)]
         {
@@ -176,6 +178,7 @@ impl IndexScheduler {
                     op,
                     &progress,
                     current_batch.embedder_stats.clone(),
+                    network,
                 )?;
 
                 {
@@ -235,6 +238,7 @@ impl IndexScheduler {
                     Batch::IndexUpdate { index_uid, primary_key, new_index_uid: None, task },
                     current_batch,
                     progress,
+                    network,
                 )
             }
             Batch::IndexUpdate { index_uid, primary_key, new_index_uid, mut task } => {
@@ -539,6 +543,10 @@ impl IndexScheduler {
 
                 Ok((tasks, ProcessBatchInfo::default()))
             }
+            Batch::NetworkIndexBatch { network_task, inner_batch } => {
+                self.process_network_index_batch(network_task, inner_batch, current_batch, progress)
+            }
+            Batch::NetworkReady { task } => self.process_network_ready(task, progress),
         }
     }
 

crates/index-scheduler/src/scheduler/process_export.rs

@@ -1,5 +1,6 @@
 use std::collections::BTreeMap;
 use std::io::{self, Write as _};
+use std::ops::ControlFlow;
 use std::sync::atomic;
 use std::time::Duration;
 
@@ -7,6 +8,7 @@ use backoff::ExponentialBackoff;
 use byte_unit::Byte;
 use flate2::write::GzEncoder;
 use flate2::Compression;
+use meilisearch_types::error::Code;
 use meilisearch_types::index_uid_pattern::IndexUidPattern;
 use meilisearch_types::milli::constants::RESERVED_VECTORS_FIELD_NAME;
 use meilisearch_types::milli::index::EmbeddingsWithMetadata;
@@ -15,7 +17,10 @@ use meilisearch_types::milli::update::{request_threads, Setting};
 use meilisearch_types::milli::vector::parsed_vectors::{ExplicitVectors, VectorOrArrayOfVectors};
 use meilisearch_types::milli::{self, obkv_to_json, Filter, InternalError};
 use meilisearch_types::settings::{self, SecretPolicy};
+use meilisearch_types::tasks::network::headers::SetHeader as _;
+use meilisearch_types::tasks::network::{headers, ImportData, ImportMetadata, Origin};
 use meilisearch_types::tasks::{DetailsExportIndexSettings, ExportIndexSettings};
+use roaring::RoaringBitmap;
 use serde::Deserialize;
 use ureq::{json, Response};
 
@@ -50,6 +55,7 @@ impl IndexScheduler {
         let agent = ureq::AgentBuilder::new().timeout(Duration::from_secs(5)).build();
         let must_stop_processing = self.scheduler.must_stop_processing.clone();
         for (i, (_pattern, uid, export_settings)) in indexes.iter().enumerate() {
+            let err = |err| Error::from_milli(err, Some(uid.to_string()));
             if must_stop_processing.get() {
                 return Err(Error::AbortedTask);
             }
@@ -61,261 +67,474 @@ impl IndexScheduler {
             ));
 
             let ExportIndexSettings { filter, override_settings } = export_settings;
+
             let index = self.index(uid)?;
             let index_rtxn = index.read_txn()?;
-            let bearer = api_key.map(|api_key| format!("Bearer {api_key}"));
+            let filter = filter.as_ref().map(Filter::from_json).transpose().map_err(err)?.flatten();
-
+            let filter_universe =
-            // First, check if the index already exists
+                filter.map(|f| f.evaluate(&index_rtxn, &index)).transpose().map_err(err)?;
-            let url = format!("{base_url}/indexes/{uid}");
+            let whole_universe =
-            let response = retry(&must_stop_processing, || {
+                index.documents_ids(&index_rtxn).map_err(milli::Error::from).map_err(err)?;
-                let mut request = agent.get(&url);
-                if let Some(bearer) = &bearer {
-                    request = request.set("Authorization", bearer);
-                }
-
-                request.send_bytes(Default::default()).map_err(into_backoff_error)
-            });
-            let index_exists = match response {
-                Ok(response) => response.status() == 200,
-                Err(Error::FromRemoteWhenExporting { code, .. }) if code == "index_not_found" => {
-                    false
-                }
-                Err(e) => return Err(e),
-            };
-
-            let primary_key = index
-                .primary_key(&index_rtxn)
-                .map_err(|e| Error::from_milli(e.into(), Some(uid.to_string())))?;
-
-            // Create the index
-            if !index_exists {
-                let url = format!("{base_url}/indexes");
-                retry(&must_stop_processing, || {
-                    let mut request = agent.post(&url);
-                    if let Some(bearer) = &bearer {
-                        request = request.set("Authorization", bearer);
-                    }
-                    let index_param = json!({ "uid": uid, "primaryKey": primary_key });
-                    request.send_json(&index_param).map_err(into_backoff_error)
-                })?;
-            }
-
-            // Patch the index primary key
-            if index_exists && *override_settings {
-                let url = format!("{base_url}/indexes/{uid}");
-                retry(&must_stop_processing, || {
-                    let mut request = agent.patch(&url);
-                    if let Some(bearer) = &bearer {
-                        request = request.set("Authorization", bearer);
-                    }
-                    let index_param = json!({ "primaryKey": primary_key });
-                    request.send_json(&index_param).map_err(into_backoff_error)
-                })?;
-            }
-
-            // Send the index settings
-            if !index_exists || *override_settings {
-                let mut settings =
-                    settings::settings(&index, &index_rtxn, SecretPolicy::RevealSecrets)
-                        .map_err(|e| Error::from_milli(e, Some(uid.to_string())))?;
-                // Remove the experimental chat setting if not enabled
-                if self.features().check_chat_completions("exporting chat settings").is_err() {
-                    settings.chat = Setting::NotSet;
-                }
-                // Retry logic for sending settings
-                let url = format!("{base_url}/indexes/{uid}/settings");
-                retry(&must_stop_processing, || {
-                    let mut request = agent.patch(&url);
-                    if let Some(bearer) = bearer.as_ref() {
-                        request = request.set("Authorization", bearer);
-                    }
-                    request.send_json(settings.clone()).map_err(into_backoff_error)
-                })?;
-            }
-
-            let filter = filter
-                .as_ref()
-                .map(Filter::from_json)
-                .transpose()
-                .map_err(|e| Error::from_milli(e, Some(uid.to_string())))?
-                .flatten();
-
-            let filter_universe = filter
-                .map(|f| f.evaluate(&index_rtxn, &index))
-                .transpose()
-                .map_err(|e| Error::from_milli(e, Some(uid.to_string())))?;
-            let whole_universe = index
-                .documents_ids(&index_rtxn)
-                .map_err(|e| Error::from_milli(e.into(), Some(uid.to_string())))?;
             let universe = filter_universe.unwrap_or(whole_universe);
-
+            let target = TargetInstance { remote_name: None, base_url, api_key };
-            let fields_ids_map = index.fields_ids_map(&index_rtxn)?;
+            let ctx = ExportContext {
-            let all_fields: Vec<_> = fields_ids_map.iter().map(|(id, _)| id).collect();
+                index: &index,
-
+                index_rtxn: &index_rtxn,
-            // We don't need to keep this one alive as we will
+                universe: &universe,
-            // spawn many threads to process the documents
+                progress: &progress,
-            drop(index_rtxn);
+                agent: &agent,
-
+                must_stop_processing: &must_stop_processing,
-            let total_documents = universe.len() as u32;
+            };
-            let (step, progress_step) = AtomicDocumentStep::new(total_documents);
+            let options = ExportOptions {
-            progress.update_progress(progress_step);
+                index_uid: uid,
+                payload_size,
+                override_settings: *override_settings,
+                export_mode: ExportMode::ExportRoute,
+            };
+            let total_documents = self.export_one_index(target, options, ctx)?;
 
             output.insert(
                 IndexUidPattern::new_unchecked(uid.clone()),
                 DetailsExportIndexSettings {
                     settings: (*export_settings).clone(),
-                    matched_documents: Some(total_documents as u64),
+                    matched_documents: Some(total_documents),
                 },
             );
-
-            let limit = payload_size.map(|ps| ps.as_u64() as usize).unwrap_or(20 * 1024 * 1024); // defaults to 20 MiB
-            let documents_url = format!("{base_url}/indexes/{uid}/documents");
-
-            let results = request_threads()
-                .broadcast(|ctx| {
-                    let index_rtxn = index
-                        .read_txn()
-                        .map_err(|e| Error::from_milli(e.into(), Some(uid.to_string())))?;
-
-                    let mut buffer = Vec::new();
-                    let mut tmp_buffer = Vec::new();
-                    let mut compressed_buffer = Vec::new();
-                    for (i, docid) in universe.iter().enumerate() {
-                        if i % ctx.num_threads() != ctx.index() {
-                            continue;
-                        }
-
-                        let document = index
-                            .document(&index_rtxn, docid)
-                            .map_err(|e| Error::from_milli(e, Some(uid.to_string())))?;
-
-                        let mut document = obkv_to_json(&all_fields, &fields_ids_map, document)
-                            .map_err(|e| Error::from_milli(e, Some(uid.to_string())))?;
-
-                        // TODO definitely factorize this code
-                        'inject_vectors: {
-                            let embeddings = index
-                                .embeddings(&index_rtxn, docid)
-                                .map_err(|e| Error::from_milli(e, Some(uid.to_string())))?;
-
-                            if embeddings.is_empty() {
-                                break 'inject_vectors;
-                            }
-
-                            let vectors = document
-                                .entry(RESERVED_VECTORS_FIELD_NAME)
-                                .or_insert(serde_json::Value::Object(Default::default()));
-
-                            let serde_json::Value::Object(vectors) = vectors else {
-                                return Err(Error::from_milli(
-                                    milli::Error::UserError(
-                                        milli::UserError::InvalidVectorsMapType {
-                                            document_id: {
-                                                if let Ok(Some(Ok(index))) = index
-                                                    .external_id_of(
-                                                        &index_rtxn,
-                                                        std::iter::once(docid),
-                                                    )
-                                                    .map(|it| it.into_iter().next())
-                                                {
-                                                    index
-                                                } else {
-                                                    format!("internal docid={docid}")
-                                                }
-                                            },
-                                            value: vectors.clone(),
-                                        },
-                                    ),
-                                    Some(uid.to_string()),
-                                ));
-                            };
-
-                            for (
-                                embedder_name,
-                                EmbeddingsWithMetadata { embeddings, regenerate, has_fragments },
-                            ) in embeddings
-                            {
-                                let embeddings = ExplicitVectors {
-                                    embeddings: Some(
-                                        VectorOrArrayOfVectors::from_array_of_vectors(embeddings),
-                                    ),
-                                    regenerate: regenerate &&
-                                    // Meilisearch does not handle well dumps with fragments, because as the fragments
-                                    // are marked as user-provided,
-                                    // all embeddings would be regenerated on any settings change or document update.
-                                    // To prevent this, we mark embeddings has non regenerate in this case.
-                                    !has_fragments,
-                                };
-                                vectors.insert(
-                                    embedder_name,
-                                    serde_json::to_value(embeddings).unwrap(),
-                                );
-                            }
-                        }
-
-                        tmp_buffer.clear();
-                        serde_json::to_writer(&mut tmp_buffer, &document)
-                            .map_err(milli::InternalError::from)
-                            .map_err(|e| Error::from_milli(e.into(), Some(uid.to_string())))?;
-
-                        // Make sure we put at least one document in the buffer even
-                        // though we might go above the buffer limit before sending
-                        if !buffer.is_empty() && buffer.len() + tmp_buffer.len() > limit {
-                            // We compress the documents before sending them
-                            let mut encoder =
-                                GzEncoder::new(&mut compressed_buffer, Compression::default());
-                            encoder
-                                .write_all(&buffer)
-                                .map_err(|e| Error::from_milli(e.into(), Some(uid.clone())))?;
-                            encoder
-                                .finish()
-                                .map_err(|e| Error::from_milli(e.into(), Some(uid.clone())))?;
-
-                            retry(&must_stop_processing, || {
-                                let mut request = agent.post(&documents_url);
-                                request = request.set("Content-Type", "application/x-ndjson");
-                                request = request.set("Content-Encoding", "gzip");
-                                if let Some(bearer) = &bearer {
-                                    request = request.set("Authorization", bearer);
-                                }
-                                request.send_bytes(&compressed_buffer).map_err(into_backoff_error)
-                            })?;
-                            buffer.clear();
-                            compressed_buffer.clear();
-                        }
-                        buffer.extend_from_slice(&tmp_buffer);
-
-                        if i > 0 && i % 100 == 0 {
-                            step.fetch_add(100, atomic::Ordering::Relaxed);
-                        }
-                    }
-
-                    retry(&must_stop_processing, || {
-                        let mut request = agent.post(&documents_url);
-                        request = request.set("Content-Type", "application/x-ndjson");
-                        if let Some(bearer) = &bearer {
-                            request = request.set("Authorization", bearer);
-                        }
-                        request.send_bytes(&buffer).map_err(into_backoff_error)
-                    })?;
-
-                    Ok(())
-                })
-                .map_err(|e| {
-                    Error::from_milli(
-                        milli::Error::InternalError(InternalError::PanicInThreadPool(e)),
-                        Some(uid.to_string()),
-                    )
-                })?;
-            for result in results {
-                result?;
-            }
-
-            step.store(total_documents, atomic::Ordering::Relaxed);
         }
 
         Ok(output)
     }
+
+    pub(super) fn export_one_index(
+        &self,
+        target: TargetInstance<'_>,
+        options: ExportOptions<'_>,
+        ctx: ExportContext<'_>,
+    ) -> Result<u64, Error> {
+        let err = |err| Error::from_milli(err, Some(options.index_uid.to_string()));
+        let total_index_documents = ctx.universe.len();
+        let task_network = options.task_network(total_index_documents);
+
+        let bearer = target.api_key.map(|api_key| format!("Bearer {api_key}"));
+        let url = format!(
+            "{base_url}/indexes/{index_uid}",
+            base_url = target.base_url,
+            index_uid = options.index_uid
+        );
+        let response = retry(ctx.must_stop_processing, || {
+            let mut request = ctx.agent.get(&url);
+            if let Some(bearer) = &bearer {
+                request = request.set("Authorization", bearer);
+            }
+
+            request.send_bytes(Default::default()).map_err(into_backoff_error)
+        });
+        let index_exists = match response {
+            Ok(response) => response.status() == 200,
+            Err(Error::FromRemoteWhenExporting { code, .. })
+                if code == Code::IndexNotFound.name() =>
+            {
+                false
+            }
+            Err(e) => return Err(e),
+        };
+        let primary_key =
+            ctx.index.primary_key(ctx.index_rtxn).map_err(milli::Error::from).map_err(err)?;
+        if !index_exists {
+            let url = format!("{base_url}/indexes", base_url = target.base_url);
+            let _ = handle_response(
+                target.remote_name,
+                retry(ctx.must_stop_processing, || {
+                    let mut request = ctx.agent.post(&url);
+
+                    if let Some((import_data, origin, metadata)) = &task_network {
+                        request = set_network_ureq_headers(request, import_data, origin, metadata);
+                    }
+
+                    if let Some(bearer) = bearer.as_ref() {
+                        request = request.set("Authorization", bearer);
+                    }
+                    let index_param =
+                        json!({ "uid": options.index_uid, "primaryKey": primary_key });
+
+                    request.send_json(&index_param).map_err(into_backoff_error)
+                }),
+            )?;
+        }
+        if index_exists && options.override_settings {
+            let _ = handle_response(
+                target.remote_name,
+                retry(ctx.must_stop_processing, || {
+                    let mut request = ctx.agent.patch(&url);
+                    if let Some((import_data, origin, metadata)) = &task_network {
+                        request = set_network_ureq_headers(request, import_data, origin, metadata);
+                    }
+                    if let Some(bearer) = &bearer {
+                        request = request.set("Authorization", bearer);
+                    }
+                    let index_param = json!({ "primaryKey": primary_key });
+                    request.send_json(&index_param).map_err(into_backoff_error)
+                }),
+            )?;
+        }
+        if !index_exists || options.override_settings {
+            let mut settings =
+                settings::settings(ctx.index, ctx.index_rtxn, SecretPolicy::RevealSecrets)
+                    .map_err(err)?;
+            // Remove the experimental chat setting if not enabled
+            if self.features().check_chat_completions("exporting chat settings").is_err() {
+                settings.chat = Setting::NotSet;
+            }
+            // Retry logic for sending settings
+            let url = format!(
+                "{base_url}/indexes/{index_uid}/settings",
+                base_url = target.base_url,
+                index_uid = options.index_uid
+            );
+
+            let _ = handle_response(
+                target.remote_name,
+                retry(ctx.must_stop_processing, || {
+                    let mut request = ctx.agent.patch(&url);
+
+                    if let Some((import_data, origin, metadata)) = &task_network {
+                        request = set_network_ureq_headers(request, import_data, origin, metadata);
+                    }
+
+                    if let Some(bearer) = bearer.as_ref() {
+                        request = request.set("Authorization", bearer);
+                    }
+                    request.send_json(settings.clone()).map_err(into_backoff_error)
+                }),
+            )?;
+        }
+
+        let fields_ids_map = ctx.index.fields_ids_map(ctx.index_rtxn)?;
+        let all_fields: Vec<_> = fields_ids_map.iter().map(|(id, _)| id).collect();
+        let total_documents = ctx.universe.len() as u32;
+        let (step, progress_step) = AtomicDocumentStep::new(total_documents);
+        ctx.progress.update_progress(progress_step);
+
+        let limit = options
+            .payload_size
+            .map(|ps| ps.as_u64() as usize)
+            .unwrap_or(self.export_default_payload_size_bytes.as_u64() as usize);
+        let documents_url = format!(
+            "{base_url}/indexes/{index_uid}/documents",
+            base_url = target.base_url,
+            index_uid = options.index_uid
+        );
+
+        // no document to send, but we must still send a task when performing network balancing
+        if ctx.universe.is_empty() {
+            if let Some((import_data, network_change_origin, metadata)) = task_network {
+                let mut compressed_buffer = Vec::new();
+                // ignore control flow, we're returning anyway
+                let _ = send_buffer(
+                    b" ", // needs something otherwise meili complains about missing payload
+                    &mut compressed_buffer,
+                    ctx.must_stop_processing,
+                    ctx.agent,
+                    &documents_url,
+                    target.remote_name,
+                    bearer.as_deref(),
+                    Some(&(import_data, network_change_origin.clone(), metadata)),
+                    &err,
+                )?;
+            }
+            return Ok(0);
+        }
+
+        let results = request_threads()
+            .broadcast(|broadcast| {
+                let mut task_network = options.task_network(total_index_documents);
+
+                let index_rtxn = ctx.index.read_txn().map_err(milli::Error::from).map_err(err)?;
+
+                let mut buffer = Vec::new();
+                let mut tmp_buffer = Vec::new();
+                let mut compressed_buffer = Vec::new();
+                for (i, docid) in ctx.universe.iter().enumerate() {
+                    if i % broadcast.num_threads() != broadcast.index() {
+                        continue;
+                    }
+                    if let Some((import_data, _, metadata)) = &mut task_network {
+                        import_data.document_count += 1;
+                        metadata.task_key = Some(docid);
+                    }
+
+                    let document = ctx.index.document(&index_rtxn, docid).map_err(err)?;
+
+                    let mut document =
+                        obkv_to_json(&all_fields, &fields_ids_map, document).map_err(err)?;
+
+                    // TODO definitely factorize this code
+                    'inject_vectors: {
+                        let embeddings = ctx.index.embeddings(&index_rtxn, docid).map_err(err)?;
+
+                        if embeddings.is_empty() {
+                            break 'inject_vectors;
+                        }
+
+                        let vectors = document
+                            .entry(RESERVED_VECTORS_FIELD_NAME)
+                            .or_insert(serde_json::Value::Object(Default::default()));
+
+                        let serde_json::Value::Object(vectors) = vectors else {
+                            return Err(err(milli::Error::UserError(
+                                milli::UserError::InvalidVectorsMapType {
+                                    document_id: {
+                                        if let Ok(Some(Ok(index))) = ctx
+                                            .index
+                                            .external_id_of(&index_rtxn, std::iter::once(docid))
+                                            .map(|it| it.into_iter().next())
+                                        {
+                                            index
+                                        } else {
+                                            format!("internal docid={docid}")
+                                        }
+                                    },
+                                    value: vectors.clone(),
+                                },
+                            )));
+                        };
+
+                        for (
+                            embedder_name,
+                            EmbeddingsWithMetadata { embeddings, regenerate, has_fragments },
+                        ) in embeddings
+                        {
+                            let embeddings = ExplicitVectors {
+                                embeddings: Some(VectorOrArrayOfVectors::from_array_of_vectors(
+                                    embeddings,
+                                )),
+                                regenerate: regenerate &&
+                                // Meilisearch does not handle well dumps with fragments, because as the fragments
+                                // are marked as user-provided,
+                                // all embeddings would be regenerated on any settings change or document update.
+                                // To prevent this, we mark embeddings has non regenerate in this case.
+                                !has_fragments,
+                            };
+                            vectors
+                                .insert(embedder_name, serde_json::to_value(embeddings).unwrap());
+                        }
+                    }
+
+                    tmp_buffer.clear();
+                    serde_json::to_writer(&mut tmp_buffer, &document)
+                        .map_err(milli::InternalError::from)
+                        .map_err(milli::Error::from)
+                        .map_err(err)?;
+
+                    // Make sure we put at least one document in the buffer even
+                    // though we might go above the buffer limit before sending
+                    if !buffer.is_empty() && buffer.len() + tmp_buffer.len() > limit {
+                        let control_flow = send_buffer(
+                            &buffer,
+                            &mut compressed_buffer,
+                            ctx.must_stop_processing,
+                            ctx.agent,
+                            &documents_url,
+                            target.remote_name,
+                            bearer.as_deref(),
+                            task_network.as_ref(),
+                            &err,
+                        )?;
+                        buffer.clear();
+                        compressed_buffer.clear();
+                        if let Some((import_data, _, metadata)) = &mut task_network {
+                            import_data.document_count = 0;
+                            metadata.task_key = None;
+                        }
+                        if control_flow.is_break() {
+                            return Ok(());
+                        }
+                    }
+                    buffer.extend_from_slice(&tmp_buffer);
+
+                    if i > 0 && i % 100 == 0 {
+                        step.fetch_add(100, atomic::Ordering::Relaxed);
+                    }
+                }
+
+                // send the last buffered documents if any
+                if !buffer.is_empty() {
+                    // ignore control flow here
+                    let _ = send_buffer(
+                        &buffer,
+                        &mut compressed_buffer,
+                        ctx.must_stop_processing,
+                        ctx.agent,
+                        &documents_url,
+                        target.remote_name,
+                        bearer.as_deref(),
+                        task_network.as_ref(),
+                        &err,
+                    )?;
+                }
+
+                Ok(())
+            })
+            .map_err(|e| err(milli::Error::InternalError(InternalError::PanicInThreadPool(e))))?;
+        for result in results {
+            result?;
+        }
+        step.store(total_documents, atomic::Ordering::Relaxed);
+        Ok(total_documents as u64)
+    }
+
+    #[cfg(feature = "enterprise")] // only used in enterprise edition for now
+    pub(super) fn export_no_index(
+        &self,
+        target: TargetInstance<'_>,
+        export_old_remote_name: &str,
+        network_change_origin: &Origin,
+        agent: &ureq::Agent,
+        must_stop_processing: &MustStopProcessing,
+    ) -> Result<(), Error> {
+        let bearer = target.api_key.map(|api_key| format!("Bearer {api_key}"));
+        let url = format!("{base_url}/network", base_url = target.base_url,);
+
+        {
+            let _ = handle_response(
+                target.remote_name,
+                retry(must_stop_processing, || {
+                    let request = agent.patch(&url);
+                    let mut request = set_network_ureq_headers(
+                        request,
+                        &ImportData {
+                            remote_name: export_old_remote_name.to_string(),
+                            index_name: None,
+                            document_count: 0,
+                        },
+                        network_change_origin,
+                        &ImportMetadata {
+                            index_count: 0,
+                            task_key: None,
+                            total_index_documents: 0,
+                        },
+                    );
+                    request = request.set("Content-Type", "application/json");
+                    if let Some(bearer) = &bearer {
+                        request = request.set("Authorization", bearer);
+                    }
+                    request
+                        .send_json(
+                            // empty payload that will be disregarded
+                            serde_json::Value::Object(Default::default()),
+                        )
+                        .map_err(into_backoff_error)
+                }),
+            )?;
+        }
+
+        Ok(())
+    }
+}
+
+fn set_network_ureq_headers(
+    request: ureq::Request,
+    import_data: &ImportData,
+    origin: &Origin,
+    metadata: &ImportMetadata,
+) -> ureq::Request {
+    let request = RequestWrapper(request);
+
+    let ImportMetadata { index_count, task_key, total_index_documents } = metadata;
+    let Origin { remote_name: origin_remote, task_uid, network_version } = origin;
+    let ImportData { remote_name: import_remote, index_name, document_count } = import_data;
+
+    let request = request
+        .set_origin_remote(origin_remote)
+        .set_origin_task_uid(*task_uid)
+        .set_origin_network_version(*network_version)
+        .set_import_remote(import_remote)
+        .set_import_docs(*document_count)
+        .set_import_index_count(*index_count)
+        .set_import_index_docs(*total_index_documents);
+
+    let request = if let Some(index_name) = index_name.as_deref() {
+        request.set_import_index(index_name)
+    } else {
+        request
+    };
+    let RequestWrapper(request) = if let Some(task_key) = task_key {
+        request.set_import_task_key(*task_key)
+    } else {
+        request
+    };
+
+    request
+}
+
+struct RequestWrapper(ureq::Request);
+impl headers::SetHeader for RequestWrapper {
+    fn set_header(self, name: &str, value: &str) -> Self {
+        Self(self.0.set(name, value))
+    }
+}
+
+#[allow(clippy::too_many_arguments)]
+fn send_buffer<'a>(
+    buffer: &'a [u8],
+    mut compressed_buffer: &'a mut Vec<u8>,
+    must_stop_processing: &MustStopProcessing,
+    agent: &ureq::Agent,
+    documents_url: &'a str,
+    remote_name: Option<&str>,
+    bearer: Option<&'a str>,
+    task_network: Option<&(ImportData, Origin, ImportMetadata)>,
+    err: &'a impl Fn(milli::Error) -> crate::Error,
+) -> Result<ControlFlow<(), ()>> {
+    // We compress the documents before sending them
+    let mut encoder: GzEncoder<&mut &mut Vec<u8>> =
+        GzEncoder::new(&mut compressed_buffer, Compression::default());
+    encoder.write_all(buffer).map_err(milli::Error::from).map_err(err)?;
+    encoder.finish().map_err(milli::Error::from).map_err(err)?;
+
+    let res = retry(must_stop_processing, || {
+        let mut request = agent.post(documents_url);
+        request = request.set("Content-Type", "application/x-ndjson");
+        request = request.set("Content-Encoding", "gzip");
+        if let Some(bearer) = bearer {
+            request = request.set("Authorization", bearer);
+        }
+        if let Some((import_data, origin, metadata)) = task_network {
+            request = set_network_ureq_headers(request, import_data, origin, metadata);
+        }
+        request.send_bytes(compressed_buffer).map_err(into_backoff_error)
+    });
+
+    handle_response(remote_name, res)
+}
+
+fn handle_response(remote_name: Option<&str>, res: Result<Response>) -> Result<ControlFlow<()>> {
+    let remote_name = remote_name.unwrap_or("unnamed");
+    match res {
+        Ok(_response) => Ok(ControlFlow::Continue(())),
+        Err(Error::FromRemoteWhenExporting { code, .. })
+            if code == Code::ImportTaskAlreadyReceived.name() =>
+        {
+            Ok(ControlFlow::Continue(()))
+        }
+        Err(Error::FromRemoteWhenExporting { code, message, .. })
+            if code == Code::ImportTaskUnknownRemote.name() =>
+        {
+            tracing::warn!("remote `{remote_name}` answered with: {message}");
+            Ok(ControlFlow::Break(()))
+        }
+        // note: there has already been many attempts to get this due to exponential backoff
+        Err(Error::FromRemoteWhenExporting { code, message, .. })
+            if code == Code::ImportTaskWithoutNetworkTask.name() =>
+        {
+            tracing::warn!("remote `{remote_name}` answered with: {message}");
+            Ok(ControlFlow::Break(()))
+        }
+        Err(e) => {
+            tracing::warn!("error while exporting: {e}");
+            Err(e)
+        }
+    }
 }
 
 fn retry<F>(must_stop_processing: &MustStopProcessing, send_request: F) -> Result<ureq::Response>
@@ -374,4 +593,65 @@ fn ureq_error_into_error(error: ureq::Error) -> Error {
     }
 }
 
+// export_one_index arguments
+pub(super) struct TargetInstance<'a> {
+    pub(super) remote_name: Option<&'a str>,
+    pub(super) base_url: &'a str,
+    pub(super) api_key: Option<&'a str>,
+}
+
+pub(super) struct ExportOptions<'a> {
+    pub(super) index_uid: &'a str,
+    pub(super) payload_size: Option<&'a Byte>,
+    pub(super) override_settings: bool,
+    pub(super) export_mode: ExportMode<'a>,
+}
+
+impl ExportOptions<'_> {
+    fn task_network(
+        &self,
+        total_index_documents: u64,
+    ) -> Option<(ImportData, Origin, ImportMetadata)> {
+        if let ExportMode::NetworkBalancing {
+            index_count,
+            export_old_remote_name,
+            network_change_origin,
+        } = self.export_mode
+        {
+            Some((
+                ImportData {
+                    remote_name: export_old_remote_name.to_string(),
+                    index_name: Some(self.index_uid.to_string()),
+                    document_count: 0,
+                },
+                network_change_origin.clone(),
+                ImportMetadata { index_count, task_key: None, total_index_documents },
+            ))
+        } else {
+            None
+        }
+    }
+}
+
+pub(super) struct ExportContext<'a> {
+    pub(super) index: &'a meilisearch_types::milli::Index,
+    pub(super) index_rtxn: &'a milli::heed::RoTxn<'a>,
+    pub(super) universe: &'a RoaringBitmap,
+    pub(super) progress: &'a Progress,
+    pub(super) agent: &'a ureq::Agent,
+    pub(super) must_stop_processing: &'a MustStopProcessing,
+}
+
+pub(super) enum ExportMode<'a> {
+    ExportRoute,
+    #[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
+    NetworkBalancing {
+        index_count: u64,
+
+        export_old_remote_name: &'a str,
+        network_change_origin: &'a Origin,
+    },
+}
+
+// progress related
 enum ExportIndex {}

crates/index-scheduler/src/scheduler/process_index_operation.rs

@@ -8,6 +8,7 @@ use meilisearch_types::milli::progress::{EmbedderStats, Progress};
 use meilisearch_types::milli::update::new::indexer::{self, UpdateByFunction};
 use meilisearch_types::milli::update::DocumentAdditionResult;
 use meilisearch_types::milli::{self, ChannelCongestion, Filter};
+use meilisearch_types::network::Network;
 use meilisearch_types::settings::apply_settings_to_builder;
 use meilisearch_types::tasks::{Details, KindWithContent, Status, Task};
 use meilisearch_types::Index;
@@ -36,6 +37,7 @@ impl IndexScheduler {
         operation: IndexOperation,
         progress: &Progress,
         embedder_stats: Arc<EmbedderStats>,
+        network: &Network,
     ) -> Result<(Vec<Task>, Option<ChannelCongestion>)> {
         let indexer_alloc = Bump::new();
         let started_processing_at = std::time::Instant::now();
@@ -67,8 +69,6 @@ impl IndexScheduler {
             IndexOperation::DocumentOperation { index_uid, primary_key, operations, mut tasks } => {
                 progress.update_progress(DocumentOperationProgress::RetrievingConfig);
 
-                let network = self.network();
-
                 let shards = network.shards();
 
                 // TODO: at some point, for better efficiency we might want to reuse the bumpalo for successive batches.
@@ -77,8 +77,8 @@ impl IndexScheduler {
                 let mut content_files = Vec::new();
                 for operation in &operations {
                     match operation {
-                        DocumentOperation::Replace(content_uuid)
+                        DocumentOperation::Replace { content_file: content_uuid, .. }
-                        | DocumentOperation::Update(content_uuid) => {
+                        | DocumentOperation::Update { content_file: content_uuid, .. } => {
                             let content_file = self.queue.file_store.get_update(*content_uuid)?;
                             let mmap = unsafe { memmap2::Mmap::map(&content_file)? };
                             content_files.push(mmap);
@@ -100,16 +100,16 @@ impl IndexScheduler {
                 let embedders = self.embedders(index_uid.clone(), embedders)?;
                 for operation in operations {
                     match operation {
-                        DocumentOperation::Replace(_content_uuid) => {
+                        DocumentOperation::Replace { content_file: _, on_missing_document } => {
                             let mmap = content_files_iter.next().unwrap();
                             indexer
-                                .replace_documents(mmap)
+                                .replace_documents(mmap, on_missing_document)
                                 .map_err(|e| Error::from_milli(e, Some(index_uid.clone())))?;
                         }
-                        DocumentOperation::Update(_content_uuid) => {
+                        DocumentOperation::Update { content_file: _, on_missing_document } => {
                             let mmap = content_files_iter.next().unwrap();
                             indexer
-                                .update_documents(mmap)
+                                .update_documents(mmap, on_missing_document)
                                 .map_err(|e| Error::from_milli(e, Some(index_uid.clone())))?;
                         }
                         DocumentOperation::Delete(document_ids) => {
@@ -504,6 +504,7 @@ impl IndexScheduler {
                     },
                     progress,
                     embedder_stats.clone(),
+                    network,
                 )?;
 
                 let (settings_tasks, _congestion) = self.apply_index_operation(
@@ -512,6 +513,7 @@ impl IndexScheduler {
                     IndexOperation::Settings { index_uid, settings, tasks: settings_tasks },
                     progress,
                     embedder_stats,
+                    network,
                 )?;
 
                 let mut tasks = settings_tasks;

crates/index-scheduler/src/scheduler/process_snapshot_creation.rs

@@ -12,7 +12,7 @@ use crate::processing::{AtomicUpdateFileStep, SnapshotCreationProgress};
 use crate::queue::TaskQueue;
 use crate::{Error, IndexScheduler, Result};
 
-const UPDATE_FILES_DIR_NAME: &str = "update_files";
+pub(crate) const UPDATE_FILES_DIR_NAME: &str = "update_files";
 
 /// # Safety
 ///
@@ -230,407 +230,4 @@ impl IndexScheduler {
 
         Ok(tasks)
     }
-
-    #[cfg(unix)]
-    pub(super) async fn process_snapshot_to_s3(
-        &self,
-        progress: Progress,
-        opts: meilisearch_types::milli::update::S3SnapshotOptions,
-        mut tasks: Vec<Task>,
-    ) -> Result<Vec<Task>> {
-        use meilisearch_types::milli::update::S3SnapshotOptions;
-
-        let S3SnapshotOptions {
-            s3_bucket_url,
-            s3_bucket_region,
-            s3_bucket_name,
-            s3_snapshot_prefix,
-            s3_access_key,
-            s3_secret_key,
-            s3_max_in_flight_parts,
-            s3_compression_level: level,
-            s3_signature_duration,
-            s3_multipart_part_size,
-        } = opts;
-
-        let must_stop_processing = self.scheduler.must_stop_processing.clone();
-        let retry_backoff = backoff::ExponentialBackoff::default();
-        let db_name = {
-            let mut base_path = self.env.path().to_owned();
-            base_path.pop();
-            base_path.file_name().and_then(OsStr::to_str).unwrap_or("data.ms").to_string()
-        };
-
-        let (reader, writer) = std::io::pipe()?;
-        let uploader_task = tokio::spawn(multipart_stream_to_s3(
-            s3_bucket_url,
-            s3_bucket_region,
-            s3_bucket_name,
-            s3_snapshot_prefix,
-            s3_access_key,
-            s3_secret_key,
-            s3_max_in_flight_parts,
-            s3_signature_duration,
-            s3_multipart_part_size,
-            must_stop_processing,
-            retry_backoff,
-            db_name,
-            reader,
-        ));
-
-        let index_scheduler = IndexScheduler::private_clone(self);
-        let builder_task = tokio::task::spawn_blocking(move || {
-            stream_tarball_into_pipe(progress, level, writer, index_scheduler)
-        });
-
-        let (uploader_result, builder_result) = tokio::join!(uploader_task, builder_task);
-
-        // Check uploader result first to early return on task abortion.
-        // safety: JoinHandle can return an error if the task was aborted, cancelled, or panicked.
-        uploader_result.unwrap()?;
-        builder_result.unwrap()?;
-
-        for task in &mut tasks {
-            task.status = Status::Succeeded;
-        }
-
-        Ok(tasks)
-    }
-}
-
-/// Streams a tarball of the database content into a pipe.
-#[cfg(unix)]
-fn stream_tarball_into_pipe(
-    progress: Progress,
-    level: u32,
-    writer: std::io::PipeWriter,
-    index_scheduler: IndexScheduler,
-) -> std::result::Result<(), Error> {
-    use std::io::Write as _;
-    use std::path::Path;
-
-    let writer = flate2::write::GzEncoder::new(writer, flate2::Compression::new(level));
-    let mut tarball = tar::Builder::new(writer);
-
-    // 1. Snapshot the version file
-    tarball
-        .append_path_with_name(&index_scheduler.scheduler.version_file_path, VERSION_FILE_NAME)?;
-
-    // 2. Snapshot the index scheduler LMDB env
-    progress.update_progress(SnapshotCreationProgress::SnapshotTheIndexScheduler);
-    let tasks_env_file = index_scheduler.env.try_clone_inner_file()?;
-    let path = Path::new("tasks").join("data.mdb");
-    append_file_to_tarball(&mut tarball, path, tasks_env_file)?;
-
-    // 2.3 Create a read transaction on the index-scheduler
-    let rtxn = index_scheduler.env.read_txn()?;
-
-    // 2.4 Create the update files directory
-    //     And only copy the update files of the enqueued tasks
-    progress.update_progress(SnapshotCreationProgress::SnapshotTheUpdateFiles);
-    let enqueued = index_scheduler.queue.tasks.get_status(&rtxn, Status::Enqueued)?;
-    let (atomic, update_file_progress) = AtomicUpdateFileStep::new(enqueued.len() as u32);
-    progress.update_progress(update_file_progress);
-
-    // We create the update_files directory so that it
-    // always exists even if there are no update files
-    let update_files_dir = Path::new(UPDATE_FILES_DIR_NAME);
-    let src_update_files_dir = {
-        let mut path = index_scheduler.env.path().to_path_buf();
-        path.pop();
-        path.join(UPDATE_FILES_DIR_NAME)
-    };
-    tarball.append_dir(update_files_dir, src_update_files_dir)?;
-
-    for task_id in enqueued {
-        let task = index_scheduler
-            .queue
-            .tasks
-            .get_task(&rtxn, task_id)?
-            .ok_or(Error::CorruptedTaskQueue)?;
-        if let Some(content_uuid) = task.content_uuid() {
-            use std::fs::File;
-
-            let src = index_scheduler.queue.file_store.update_path(content_uuid);
-            let mut update_file = File::open(src)?;
-            let path = update_files_dir.join(content_uuid.to_string());
-            tarball.append_file(path, &mut update_file)?;
-        }
-        atomic.fetch_add(1, Ordering::Relaxed);
-    }
-
-    // 3. Snapshot every indexes
-    progress.update_progress(SnapshotCreationProgress::SnapshotTheIndexes);
-    let index_mapping = index_scheduler.index_mapper.index_mapping;
-    let nb_indexes = index_mapping.len(&rtxn)? as u32;
-    let indexes_dir = Path::new("indexes");
-    let indexes_references: Vec<_> = index_scheduler
-        .index_mapper
-        .index_mapping
-        .iter(&rtxn)?
-        .map(|res| res.map_err(Error::from).map(|(name, uuid)| (name.to_string(), uuid)))
-        .collect::<Result<_, Error>>()?;
-
-    // It's prettier to use a for loop instead of the IndexMapper::try_for_each_index
-    // method, especially when we need to access the UUID, local path and index number.
-    for (i, (name, uuid)) in indexes_references.into_iter().enumerate() {
-        progress.update_progress(VariableNameStep::<SnapshotCreationProgress>::new(
-            &name, i as u32, nb_indexes,
-        ));
-        let path = indexes_dir.join(uuid.to_string()).join("data.mdb");
-        let index = index_scheduler.index_mapper.index(&rtxn, &name)?;
-        let index_file = index.try_clone_inner_file()?;
-        tracing::trace!("Appending index file for {name} in {}", path.display());
-        append_file_to_tarball(&mut tarball, path, index_file)?;
-    }
-
-    drop(rtxn);
-
-    // 4. Snapshot the auth LMDB env
-    progress.update_progress(SnapshotCreationProgress::SnapshotTheApiKeys);
-    let auth_env_file = index_scheduler.scheduler.auth_env.try_clone_inner_file()?;
-    let path = Path::new("auth").join("data.mdb");
-    append_file_to_tarball(&mut tarball, path, auth_env_file)?;
-
-    let mut gzencoder = tarball.into_inner()?;
-    gzencoder.flush()?;
-    gzencoder.try_finish()?;
-    let mut writer = gzencoder.finish()?;
-    writer.flush()?;
-
-    Result::<_, Error>::Ok(())
-}
-
-#[cfg(unix)]
-fn append_file_to_tarball<W, P>(
-    tarball: &mut tar::Builder<W>,
-    path: P,
-    mut auth_env_file: fs::File,
-) -> Result<(), Error>
-where
-    W: std::io::Write,
-    P: AsRef<std::path::Path>,
-{
-    use std::io::{Seek as _, SeekFrom};
-
-    // Note: A previous snapshot operation may have left the cursor
-    //       at the end of the file so we need to seek to the start.
-    auth_env_file.seek(SeekFrom::Start(0))?;
-    tarball.append_file(path, &mut auth_env_file)?;
-    Ok(())
-}
-
-/// Streams the content read from the given reader to S3.
-#[cfg(unix)]
-#[allow(clippy::too_many_arguments)]
-async fn multipart_stream_to_s3(
-    s3_bucket_url: String,
-    s3_bucket_region: String,
-    s3_bucket_name: String,
-    s3_snapshot_prefix: String,
-    s3_access_key: String,
-    s3_secret_key: String,
-    s3_max_in_flight_parts: std::num::NonZero<usize>,
-    s3_signature_duration: std::time::Duration,
-    s3_multipart_part_size: u64,
-    must_stop_processing: super::MustStopProcessing,
-    retry_backoff: backoff::exponential::ExponentialBackoff<backoff::SystemClock>,
-    db_name: String,
-    reader: std::io::PipeReader,
-) -> Result<(), Error> {
-    use std::collections::VecDeque;
-    use std::io;
-    use std::os::fd::OwnedFd;
-    use std::path::PathBuf;
-
-    use bytes::{Bytes, BytesMut};
-    use reqwest::{Client, Response};
-    use rusty_s3::actions::CreateMultipartUpload;
-    use rusty_s3::{Bucket, BucketError, Credentials, S3Action as _, UrlStyle};
-    use tokio::task::JoinHandle;
-
-    let reader = OwnedFd::from(reader);
-    let reader = tokio::net::unix::pipe::Receiver::from_owned_fd(reader)?;
-    let s3_snapshot_prefix = PathBuf::from(s3_snapshot_prefix);
-    let url =
-        s3_bucket_url.parse().map_err(BucketError::ParseError).map_err(Error::S3BucketError)?;
-    let bucket = Bucket::new(url, UrlStyle::Path, s3_bucket_name, s3_bucket_region)
-        .map_err(Error::S3BucketError)?;
-    let credential = Credentials::new(s3_access_key, s3_secret_key);
-
-    // Note for the future (rust 1.91+): use with_added_extension, it's prettier
-    let object_path = s3_snapshot_prefix.join(format!("{db_name}.snapshot"));
-    // Note: It doesn't work on Windows and if a port to this platform is needed,
-    //       use the slash-path crate or similar to get the correct path separator.
-    let object = object_path.display().to_string();
-
-    let action = bucket.create_multipart_upload(Some(&credential), &object);
-    let url = action.sign(s3_signature_duration);
-
-    let client = Client::new();
-    let resp = client.post(url).send().await.map_err(Error::S3HttpError)?;
-    let status = resp.status();
-
-    let body = match resp.error_for_status_ref() {
-        Ok(_) => resp.text().await.map_err(Error::S3HttpError)?,
-        Err(_) => {
-            return Err(Error::S3Error { status, body: resp.text().await.unwrap_or_default() })
-        }
-    };
-
-    let multipart =
-        CreateMultipartUpload::parse_response(&body).map_err(|e| Error::S3XmlError(Box::new(e)))?;
-    tracing::debug!("Starting the upload of the snapshot to {object}");
-
-    // We use this bumpalo for etags strings.
-    let bump = bumpalo::Bump::new();
-    let mut etags = Vec::<&str>::new();
-    let mut in_flight = VecDeque::<(JoinHandle<reqwest::Result<Response>>, Bytes)>::with_capacity(
-        s3_max_in_flight_parts.get(),
-    );
-
-    // Part numbers start at 1 and cannot be larger than 10k
-    for part_number in 1u16.. {
-        if must_stop_processing.get() {
-            return Err(Error::AbortedTask);
-        }
-
-        let part_upload =
-            bucket.upload_part(Some(&credential), &object, part_number, multipart.upload_id());
-        let url = part_upload.sign(s3_signature_duration);
-
-        // Wait for a buffer to be ready if there are in-flight parts that landed
-        let mut buffer = if in_flight.len() >= s3_max_in_flight_parts.get() {
-            let (handle, buffer) = in_flight.pop_front().expect("At least one in flight request");
-            let resp = join_and_map_error(handle).await?;
-            extract_and_append_etag(&bump, &mut etags, resp.headers())?;
-
-            let mut buffer = match buffer.try_into_mut() {
-                Ok(buffer) => buffer,
-                Err(_) => unreachable!("All bytes references were consumed in the task"),
-            };
-            buffer.clear();
-            buffer
-        } else {
-            BytesMut::with_capacity(s3_multipart_part_size as usize)
-        };
-
-        // If we successfully read enough bytes,
-        // we can continue and send the buffer/part
-        while buffer.len() < (s3_multipart_part_size as usize / 2) {
-            // Wait for the pipe to be readable
-
-            reader.readable().await?;
-
-            match reader.try_read_buf(&mut buffer) {
-                Ok(0) => break,
-                // We read some bytes but maybe not enough
-                Ok(_) => continue,
-                // The readiness event is a false positive.
-                Err(ref e) if e.kind() == io::ErrorKind::WouldBlock => continue,
-                Err(e) => return Err(e.into()),
-            }
-        }
-
-        if buffer.is_empty() {
-            // Break the loop if the buffer is
-            // empty after we tried to read bytes
-            break;
-        }
-
-        let body = buffer.freeze();
-        tracing::trace!("Sending part {part_number}");
-        let task = tokio::spawn({
-            let client = client.clone();
-            let body = body.clone();
-            backoff::future::retry(retry_backoff.clone(), move || {
-                let client = client.clone();
-                let url = url.clone();
-                let body = body.clone();
-                async move {
-                    match client.put(url).body(body).send().await {
-                        Ok(resp) if resp.status().is_client_error() => {
-                            resp.error_for_status().map_err(backoff::Error::Permanent)
-                        }
-                        Ok(resp) => Ok(resp),
-                        Err(e) => Err(backoff::Error::transient(e)),
-                    }
-                }
-            })
-        });
-        in_flight.push_back((task, body));
-    }
-
-    for (handle, _buffer) in in_flight {
-        let resp = join_and_map_error(handle).await?;
-        extract_and_append_etag(&bump, &mut etags, resp.headers())?;
-    }
-
-    tracing::debug!("Finalizing the multipart upload");
-
-    let action = bucket.complete_multipart_upload(
-        Some(&credential),
-        &object,
-        multipart.upload_id(),
-        etags.iter().map(AsRef::as_ref),
-    );
-    let url = action.sign(s3_signature_duration);
-    let body = action.body();
-    let resp = backoff::future::retry(retry_backoff, move || {
-        let client = client.clone();
-        let url = url.clone();
-        let body = body.clone();
-        async move {
-            match client.post(url).body(body).send().await {
-                Ok(resp) if resp.status().is_client_error() => {
-                    Err(backoff::Error::Permanent(Error::S3Error {
-                        status: resp.status(),
-                        body: resp.text().await.unwrap_or_default(),
-                    }))
-                }
-                Ok(resp) => Ok(resp),
-                Err(e) => Err(backoff::Error::transient(Error::S3HttpError(e))),
-            }
-        }
-    })
-    .await?;
-
-    let status = resp.status();
-    let body = resp.text().await.map_err(|e| Error::S3Error { status, body: e.to_string() })?;
-    if status.is_success() {
-        Ok(())
-    } else {
-        Err(Error::S3Error { status, body })
-    }
-}
-
-#[cfg(unix)]
-async fn join_and_map_error(
-    join_handle: tokio::task::JoinHandle<Result<reqwest::Response, reqwest::Error>>,
-) -> Result<reqwest::Response> {
-    // safety: Panic happens if the task (JoinHandle) was aborted, cancelled, or panicked
-    let request = join_handle.await.unwrap();
-    let resp = request.map_err(Error::S3HttpError)?;
-    match resp.error_for_status_ref() {
-        Ok(_) => Ok(resp),
-        Err(_) => Err(Error::S3Error {
-            status: resp.status(),
-            body: resp.text().await.unwrap_or_default(),
-        }),
-    }
-}
-
-#[cfg(unix)]
-fn extract_and_append_etag<'b>(
-    bump: &'b bumpalo::Bump,
-    etags: &mut Vec<&'b str>,
-    headers: &reqwest::header::HeaderMap,
-) -> Result<()> {
-    use reqwest::header::ETAG;
-
-    let etag = headers.get(ETAG).ok_or_else(|| Error::S3XmlError("Missing ETag header".into()))?;
-    let etag = etag.to_str().map_err(|e| Error::S3XmlError(Box::new(e)))?;
-    etags.push(bump.alloc_str(etag));
-
-    Ok(())
 }

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_enqueued_task/cancel_processed.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: canceled, canceled_by: 1, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: canceled, canceled_by: 1, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 1 {uid: 1, batch_uid: 0, status: succeeded, details: { matched_tasks: 1, canceled_tasks: Some(1), original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_enqueued_task/initial_tasks_enqueued.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 1 {uid: 1, status: enqueued, details: { matched_tasks: 1, canceled_tasks: None, original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_mix_of_tasks/aborted_indexation.snap

@@ -7,9 +7,9 @@ source: crates/index-scheduler/src/scheduler/test.rs
 {uid: 1, details: {"receivedDocuments":1,"indexedDocuments":null}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"documentAdditionOrUpdate":1},"indexUids":{"beavero":1}}, stop reason: "batched all enqueued tasks for index `beavero`", }
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "beavero", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "beavero", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "wolfo", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 1, allow_index_creation: true }}
+2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "wolfo", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 3 {uid: 3, status: enqueued, details: { matched_tasks: 3, canceled_tasks: None, original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0, 1, 2]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_mix_of_tasks/cancel_processed.snap

@@ -6,9 +6,9 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-1 {uid: 1, batch_uid: 1, status: canceled, canceled_by: 3, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "beavero", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, batch_uid: 1, status: canceled, canceled_by: 3, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "beavero", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-2 {uid: 2, batch_uid: 1, status: canceled, canceled_by: 3, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "wolfo", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 1, allow_index_creation: true }}
+2 {uid: 2, batch_uid: 1, status: canceled, canceled_by: 3, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "wolfo", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 3 {uid: 3, batch_uid: 1, status: succeeded, details: { matched_tasks: 3, canceled_tasks: Some(2), original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0, 1, 2]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_mix_of_tasks/first_task_processed.snap

@@ -6,9 +6,9 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "beavero", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "beavero", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "wolfo", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 1, allow_index_creation: true }}
+2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "wolfo", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [1,2,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_mix_of_tasks/processing_second_task_cancel_enqueued.snap

@@ -7,9 +7,9 @@ source: crates/index-scheduler/src/scheduler/test.rs
 {uid: 1, details: {"receivedDocuments":1,"indexedDocuments":null}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"documentAdditionOrUpdate":1},"indexUids":{"beavero":1}}, stop reason: "batched all enqueued tasks for index `beavero`", }
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "beavero", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "beavero", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "wolfo", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 1, allow_index_creation: true }}
+2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "wolfo", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 3 {uid: 3, status: enqueued, details: { matched_tasks: 3, canceled_tasks: None, original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0, 1, 2]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_processing_task/aborted_indexation.snap

@@ -7,7 +7,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 {uid: 0, details: {"receivedDocuments":1,"indexedDocuments":null}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"documentAdditionOrUpdate":1},"indexUids":{"catto":1}}, stop reason: "batched all enqueued tasks", }
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 1 {uid: 1, status: enqueued, details: { matched_tasks: 1, canceled_tasks: None, original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_processing_task/cancel_processed.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: canceled, canceled_by: 1, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: canceled, canceled_by: 1, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 1 {uid: 1, batch_uid: 0, status: succeeded, details: { matched_tasks: 1, canceled_tasks: Some(1), original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_processing_task/cancel_task_registered.snap

@@ -7,7 +7,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 {uid: 0, details: {"receivedDocuments":1,"indexedDocuments":null}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"documentAdditionOrUpdate":1},"indexUids":{"catto":1}}, stop reason: "batched all enqueued tasks", }
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 1 {uid: 1, status: enqueued, details: { matched_tasks: 1, canceled_tasks: None, original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_processing_task/initial_task_processing.snap

@@ -7,7 +7,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 {uid: 0, details: {"receivedDocuments":1,"indexedDocuments":null}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"documentAdditionOrUpdate":1},"indexUids":{"catto":1}}, stop reason: "batched all enqueued tasks", }
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_processing_task/registered_the_first_task.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_succeeded_task/cancel_processed.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 1 {uid: 1, batch_uid: 1, status: succeeded, details: { matched_tasks: 1, canceled_tasks: Some(0), original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_succeeded_task/initial_task_processed.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued []

crates/index-scheduler/src/scheduler/snapshots/test.rs/cancel_succeeded_task/registered_the_first_task.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion/before_index_creation.snap

@@ -7,7 +7,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 ----------------------------------------------------------------------
 ### All Tasks:
 0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 2 {uid: 2, status: enqueued, details: { deleted_documents: None }, kind: IndexDeletion { index_uid: "doggos" }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion/both_task_succeeded.snap

@@ -7,7 +7,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 ----------------------------------------------------------------------
 ### All Tasks:
 0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, batch_uid: 1, status: succeeded, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 2 {uid: 2, batch_uid: 1, status: succeeded, details: { deleted_documents: Some(0) }, kind: IndexDeletion { index_uid: "doggos" }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion/registered_the_second_task.snap

@@ -7,7 +7,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 ----------------------------------------------------------------------
 ### All Tasks:
 0 {uid: 0, status: enqueued, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion/registered_the_third_task.snap

@@ -7,7 +7,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 ----------------------------------------------------------------------
 ### All Tasks:
 0 {uid: 0, status: enqueued, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 2 {uid: 2, status: enqueued, details: { deleted_documents: None }, kind: IndexDeletion { index_uid: "doggos" }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion_on_unexisting_index/1.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 1 {uid: 1, status: enqueued, details: { deleted_documents: None }, kind: IndexDeletion { index_uid: "doggos" }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion_on_unexisting_index/2.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 1 {uid: 1, batch_uid: 0, status: succeeded, details: { deleted_documents: Some(0) }, kind: IndexDeletion { index_uid: "doggos" }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_delete_same_task_twice/initial_tasks_enqueued.snap

@@ -1,14 +1,13 @@
 ---
 source: crates/index-scheduler/src/scheduler/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_delete_same_task_twice/initial_tasks_processed.snap

@@ -6,8 +6,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [1,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_delete_same_task_twice/task_deletion_processed.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 2 {uid: 2, batch_uid: 1, status: succeeded, details: { matched_tasks: 1, deleted_tasks: Some(1), original_filter: "test_query" }, kind: TaskDeletion { query: "test_query", tasks: RoaringBitmap<[0]> }}
 3 {uid: 3, batch_uid: 1, status: succeeded, details: { matched_tasks: 1, deleted_tasks: Some(0), original_filter: "test_query" }, kind: TaskDeletion { query: "test_query", tasks: RoaringBitmap<[0]> }}
 ----------------------------------------------------------------------

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_deleteable/after_registering_the_task_deletion.snap

@@ -6,8 +6,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 2 {uid: 2, status: enqueued, details: { matched_tasks: 1, deleted_tasks: None, original_filter: "test_query" }, kind: TaskDeletion { query: "test_query", tasks: RoaringBitmap<[0]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_deleteable/initial_tasks_enqueued.snap

@@ -1,14 +1,13 @@
 ---
 source: crates/index-scheduler/src/scheduler/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_deleteable/initial_tasks_processed.snap

@@ -6,8 +6,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [1,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_deleteable/task_deletion_processed.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 2 {uid: 2, batch_uid: 1, status: succeeded, details: { matched_tasks: 1, deleted_tasks: Some(1), original_filter: "test_query" }, kind: TaskDeletion { query: "test_query", tasks: RoaringBitmap<[0]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_undeleteable/initial_tasks_enqueued.snap

@@ -7,8 +7,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 ----------------------------------------------------------------------
 ### All Tasks:
 0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,2,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_undeleteable/task_deletion_done.snap

@@ -7,8 +7,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 ----------------------------------------------------------------------
 ### All Tasks:
 0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 3 {uid: 3, batch_uid: 0, status: succeeded, details: { matched_tasks: 2, deleted_tasks: Some(0), original_filter: "test_query" }, kind: TaskDeletion { query: "test_query", tasks: RoaringBitmap<[0, 1]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_undeleteable/task_deletion_enqueued.snap

@@ -7,8 +7,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 ----------------------------------------------------------------------
 ### All Tasks:
 0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 3 {uid: 3, status: enqueued, details: { matched_tasks: 2, deleted_tasks: None, original_filter: "test_query" }, kind: TaskDeletion { query: "test_query", tasks: RoaringBitmap<[0, 1]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/task_deletion_undeleteable/task_deletion_processing.snap

@@ -8,8 +8,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 ----------------------------------------------------------------------
 ### All Tasks:
 0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
-2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true }}
+2 {uid: 2, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 3 {uid: 3, status: enqueued, details: { matched_tasks: 2, deleted_tasks: None, original_filter: "test_query" }, kind: TaskDeletion { query: "test_query", tasks: RoaringBitmap<[0, 1]> }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test_document_addition.rs/document_addition/after_register.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test_document_addition.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/scheduler/snapshots/test_document_addition.rs/document_addition/after_the_batch_creation.snap

@@ -7,7 +7,7 @@ source: crates/index-scheduler/src/scheduler/test_document_addition.rs
 {uid: 0, details: {"receivedDocuments":1,"indexedDocuments":null}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"documentAdditionOrUpdate":1},"indexUids":{"doggos":1}}, stop reason: "batched all enqueued tasks", }
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/scheduler/snapshots/test_document_addition.rs/document_addition/once_everything_is_processed.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test_document_addition.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 1, indexed_documents: Some(1) }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued []

crates/index-scheduler/src/scheduler/snapshots/test_document_addition.rs/document_addition_and_document_deletion/after_processing_the_batch.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test_document_addition.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 3, indexed_documents: Some(3) }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 3, allow_index_creation: true }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { received_documents: 3, indexed_documents: Some(3) }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 3, allow_index_creation: true, on_missing_document: Create }}
 1 {uid: 1, batch_uid: 0, status: succeeded, details: { received_document_ids: 2, deleted_documents: Some(2) }, kind: DocumentDeletion { index_uid: "doggos", documents_ids: ["1", "2"] }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test_document_addition.rs/document_addition_and_document_deletion/registered_the_first_task.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test_document_addition.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 3, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 3, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 3, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 3, allow_index_creation: true, on_missing_document: Create }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/scheduler/snapshots/test_document_addition.rs/document_addition_and_document_deletion/registered_the_second_task.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test_document_addition.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { received_documents: 3, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 3, allow_index_creation: true }}
+0 {uid: 0, status: enqueued, details: { received_documents: 3, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 3, allow_index_creation: true, on_missing_document: Create }}
 1 {uid: 1, status: enqueued, details: { received_document_ids: 2, deleted_documents: None }, kind: DocumentDeletion { index_uid: "doggos", documents_ids: ["1", "2"] }}
 ----------------------------------------------------------------------
 ### Status: