Add mold linker

2653: Bump Swatinem/rust-cache from 1.4.0 to 2.0.0 r=curquiza a=dependabot[bot]

Bumps [Swatinem/rust-cache](https://github.com/Swatinem/rust-cache) from 1.4.0 to 2.0.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/Swatinem/rust-cache/releases">Swatinem/rust-cache's releases</a>.</em></p>
<blockquote>
<h2>v2.0.0</h2>
<ul>
<li>The action code was refactored to allow for caching multiple workspaces and
different <code>target</code> directory layouts.</li>
<li>The <code>working-directory</code> and <code>target-dir</code> input options were replaced by a
single <code>workspaces</code> option that has the form of <code>$workspace -&gt; $target</code>.</li>
<li>Support for considering <code>env-vars</code> as part of the cache key.</li>
<li>The <code>sharedKey</code> input option was renamed to <code>shared-key</code> for consistency.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/Swatinem/rust-cache/blob/master/CHANGELOG.md">Swatinem/rust-cache's changelog</a>.</em></p>
<blockquote>
<h2>2.0.0</h2>
<ul>
<li>The action code was refactored to allow for caching multiple workspaces and
different <code>target</code> directory layouts.</li>
<li>The <code>working-directory</code> and <code>target-dir</code> input options were replaced by a
single <code>workspaces</code> option that has the form of <code>$workspace -&gt; $target</code>.</li>
<li>Support for considering <code>env-vars</code> as part of the cache key.</li>
<li>The <code>sharedKey</code> input option was renamed to <code>shared-key</code> for consistency.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="6720f05bc4"><code>6720f05</code></a> 2.0.0</li>
<li><a href="5733786579"><code>5733786</code></a> rebuild</li>
<li><a href="622616010e"><code>6226160</code></a> prepare v2</li>
<li><a href="0497f9301f"><code>0497f93</code></a> improve registry cleanpu</li>
<li><a href="7b8626742a"><code>7b86267</code></a> update registry cleaning</li>
<li><a href="911d8e9e55"><code>911d8e9</code></a> test sparse registry</li>
<li><a href="875be5ce2d"><code>875be5c</code></a> bump cache</li>
<li><a href="07a2ee71bc"><code>07a2ee7</code></a> lol, dependency check was reversed</li>
<li><a href="7c190ef171"><code>7c190ef</code></a> fix actual test code ;-)</li>
<li><a href="fffd6895b2"><code>fffd689</code></a> add some more tests</li>
<li>Additional commits viewable in <a href="https://github.com/Swatinem/rust-cache/compare/v1.4.0...v2.0.0">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Swatinem/rust-cache&package-manager=github_actions&previous-version=1.4.0&new-version=2.0.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting ``@dependabot` rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- ``@dependabot` rebase` will rebase this PR
- ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it
- ``@dependabot` merge` will merge this PR after your CI passes on it
- ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it
- ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging
- ``@dependabot` reopen` will reopen this PR if it is closed
- ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

.dockerignore

@@ -1,5 +1,4 @@
 target
 Dockerfile
 .dockerignore
-.git
 .gitignore

.github/ISSUE_TEMPLATE/bug_report.md

@@ -23,16 +23,8 @@ A clear and concise description of what you expected to happen.
 **Screenshots**
 If applicable, add screenshots to help explain your problem.
 
-**Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
-
-**Smartphone (please complete the following information):**
- - Device: [e.g. iPhone6]
- - OS: [e.g. iOS8.1]
- - Browser [e.g. stock browser, safari]
- - Version [e.g. 22]
+**Meilisearch version:** [e.g. v0.20.0]
 
 **Additional context**
-Add any other context about the problem here.
+Additional information that may be relevant to the issue.
+[e.g. architecture, device, OS, browser]

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,13 @@
+contact_links:
+  - name: Language support request & feedback
+    url: https://github.com/meilisearch/product/discussions/categories/feedback-feature-proposal?discussions_q=label%3Aproduct%3Acore%3Atokenizer+category%3A%22Feedback+%26+Feature+Proposal%22
+    about:  The requests and feedback regarding Language support are not managed in this repository. Please upvote the related discussion in our dedicated product repository or open a new one if it doesn't exist.
+  - name: Feature request & feedback
+    url: https://github.com/meilisearch/product/discussions/categories/feedback-feature-proposal
+    about: The feature requests and feedback regarding the already existing features are not managed in this repository. Please open a discussion in our dedicated product repository
+  - name: Documentation issue
+    url: https://github.com/meilisearch/documentation/issues/new
+    about: For documentation issues, open an issue or a PR in the documentation repository
+  - name: Support questions & other
+    url: https://github.com/meilisearch/meilisearch/discussions/new
+    about: For any other question, open a discussion in this repository

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,20 +0,0 @@
----
-name: Feature request
-about: Suggest an idea for this project
-title: ''
-labels: ''
-assignees: ''
-
----
-
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
-
-**Describe the solution you'd like**
-A clear and concise description of what you want to happen.
-
-**Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
-
-**Additional context**
-Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/tracking-issue.md

@@ -1,40 +0,0 @@
----
-name: Tracking issue
-about: Template for a tracking issue
-title: ''
-labels: tracking-issue
-assignees: ''
-
----
-
-# Summary
-
-One paragraph to explain the feature.
-
-# Motivations
-
-Why are we doing this? What use cases does it support? What is the expected outcome?
-
-# Explanation
-
-Explain the proposal like it was the final documentation of this proposal.
-
-- What is changing for end-users.
-- How it works.
-- What is breaking?
-- Examples.
-
-# Implementation
-
-Explain the technical specificities that will need to be known or done in order to implement this proposal.
-
-## Steps
-
-Describe each step to create the feature with it's associated issue/PR.
-
-# Related
-
-- [ ] Validated by the team (@people needed)
-- [ ] Test added
-- [ ] [Documentation](https://github.com/meilisearch/documentation/issues/#xxx) //Change xxx or remove the line
-- [ ] [SDK/Integrations](https://github.com/meilisearch/integration-guides/issues/#xxx) //Change xxx or remove the line

.github/dependabot.yml

@@ -0,0 +1,13 @@
+# Set update schedule for GitHub Actions only
+
+version: 2
+updates:
+
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "monthly"
+    labels:
+      - 'skip changelog'
+      - 'dependencies'
+    rebase-strategy: disabled

.github/scripts/check-release.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+# check_tag $current_tag $file_tag $file_name
+function check_tag {
+  if [[ "$1" != "$2" ]]; then
+      echo "Error: the current tag does not match the version in $3: found $2 - expected $1"
+      ret=1
+  fi
+}
+
+ret=0
+current_tag=${GITHUB_REF#'refs/tags/v'}
+
+toml_files='*/Cargo.toml'
+for toml_file in $toml_files;
+do
+    file_tag="$(grep '^version = ' $toml_file | cut -d '=' -f 2 | tr -d '"' | tr -d ' ')"
+    check_tag $current_tag $file_tag $toml_file
+done
+
+lock_file='Cargo.lock'
+lock_tag=$(grep -A 1 'name = "meilisearch-auth"' $lock_file | grep version | cut -d '=' -f 2 | tr -d '"' | tr -d ' ')
+check_tag $current_tag $lock_tag $lock_file
+
+if [[ "$ret" -eq 0 ]] ; then
+  echo 'OK'
+fi
+exit $ret

.github/scripts/is-latest-release.sh

@@ -1,14 +1,14 @@
 #!/bin/sh
 
-# Checks if the current tag should be the latest (in terms of semver and not of release date).
-# Ex: previous tag -> v0.10.1
-#     new tag -> v0.8.12
-#     The new tag should not be the latest
-#     So it returns "false", the CI should not run for the release v0.8.2
-
-# Used in GHA in publish-docker-latest.yml
+# Was used in our CIs to publish the latest docker image. Not used anymore, will be used again when v1 and v2 will be out and we will want to maintain multiple stable versions.
 # Returns "true" or "false" (as a string) to be used in the `if` in GHA
 
+# Checks if the current tag should be the latest (in terms of semver and not of release date).
+# Ex: previous tag -> v2.1.1
+#     new tag -> v1.20.3
+#     The new tag (v1.20.3) should NOT be the latest
+#     So it returns "false", the `latest` tag should not be updated for the release v1.20.3 and still need to correspond to v2.1.1
+
 # GLOBAL
 GREP_SEMVER_REGEXP='v\([0-9]*\)[.]\([0-9]*\)[.]\([0-9]*\)$' # i.e. v[number].[number].[number]
 
@@ -74,7 +74,7 @@ semverLT() {
 # Returns the tag of the latest stable release (in terms of semver and not of release date)
 get_latest() {
     temp_file='temp_file' # temp_file needed because the grep would start before the download is over
-    curl -s 'https://api.github.com/repos/meilisearch/MeiliSearch/releases' > "$temp_file"
+    curl -s 'https://api.github.com/repos/meilisearch/meilisearch/releases' > "$temp_file"
     releases=$(cat "$temp_file" | \
         grep -E "tag_name|draft|prerelease" \
         | tr -d ',"' | cut -d ':' -f2 | tr -d ' ')

.github/workflows/README.md

@@ -1,20 +0,0 @@
-# GitHub Actions Workflow for MeiliSearch
-
-> **Note:**
-
-> - We do not use [cache](https://github.com/actions/cache) yet but we could use it to speed up CI
-
-## Workflow
-
-- On each pull request, we trigger `cargo test`.
-- On each tag, we build:
-    - the tagged Docker image and publish it to Docker Hub
-    - the binaries for MacOS, Ubuntu, and Windows
-    - the Debian package
-- On each stable release (`v*.*.*` tag):
-    - we build the `latest` Docker image and publish it to Docker Hub
-    - we publish the binary to Hombrew and Gemfury
-
-## Problems
-
-- We do not test on Windows because we are unable to make it work, there is a disk space problem.

.github/workflows/check-updated-changelog.yml

@@ -1,16 +0,0 @@
-name: Check if the CHANGELOG.md has been updated
-
-on: [pull_request]
-
-jobs:
-  check:
-    name: Test on ${{ matrix.os }}
-    if: ${{ !contains(github.event.pull_request.labels.*.name, 'ignore-changelog') }}
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v2
-    - name: Checking the CHANGELOG.md has been updated in this PR
-      run: |
-        set -e
-        git fetch origin ${{ github.base_ref }}
-        git diff --name-only origin/${{ github.base_ref }} | grep -q CHANGELOG.md

.github/workflows/coverage.yml

@@ -0,0 +1,33 @@
+---
+on:
+  workflow_dispatch:
+
+name: Execute code coverage
+
+jobs:
+  nightly-coverage:
+    runs-on: ubuntu-18.04
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: nightly
+          override: true
+      - uses: actions-rs/cargo@v1
+        with:
+          command: clean
+      - uses: actions-rs/cargo@v1
+        with:
+          command: test
+          args: --all-features --no-fail-fast
+        env:
+          CARGO_INCREMENTAL: "0"
+          RUSTFLAGS: "-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=unwind -Zpanic_abort_tests"
+      - uses: actions-rs/grcov@v0.1
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          file: ${{ steps.coverage.outputs.report }}
+          yml: ./codecov.yml
+          fail_ci_if_error: true

.github/workflows/create-issue-dependencies.yml

@@ -0,0 +1,23 @@
+name: Create issue to upgrade dependencies
+on:
+  schedule:
+    - cron: '0 0 1 */3 *'
+  workflow_dispatch:
+  
+jobs:
+  create-issue:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - name: Create an issue
+      uses: actions-ecosystem/action-create-issue@v1
+      with:
+        github_token: ${{ secrets.GITHUB_TOKEN }}
+        title: Upgrade dependencies
+        body: |
+          We need to update the dependencies of the Meilisearch repository, and, if possible, the dependencies of all the core-team repositories that Meilisearch depends on (milli, charabia, heed...).
+
+          ⚠️ This issue should only be done at the beginning of the sprint!     
+        labels: |
+          dependencies
+          maintenance

.github/workflows/flaky.yml

@@ -0,0 +1,15 @@
+name: Look for flaky tests
+on:
+  schedule:
+    - cron: "0 12 * * FRI" # every friday at 12:00PM
+
+jobs:
+  flaky:
+    runs-on: ubuntu-18.04
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Install cargo-flaky
+      run: cargo install cargo-flaky
+    - name: Run cargo flaky 100 times
+      run: cargo flaky -i 100 --release

.github/workflows/publish-binaries.yml

@@ -5,14 +5,39 @@ on:
 name: Publish binaries to release
 
 jobs:
+  check-version:
+    name: Check the version validity
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      # Check if the tag has the v<nmumber>.<number>.<number> format.
+      # If yes, it means we are publishing an official release.
+      # If no, we are releasing a RC, so no need to check the version.
+      - name: Check tag format
+        if: github.event_name != 'schedule'
+        id: check-tag-format
+        run: |
+          escaped_tag=$(printf "%q" ${{ github.ref_name }})
+
+          if [[ $escaped_tag =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+            echo ::set-output name=stable::true
+          else
+            echo ::set-output name=stable::false
+          fi
+      - name: Check release validity
+        if: steps.check-tag-format.outputs.stable == 'true'
+        run: bash .github/scripts/check-release.sh
+
   publish:
-    name: Publish for ${{ matrix.os }}
+    name: Publish binary for ${{ matrix.os }}
     runs-on: ${{ matrix.os }}
+    needs: check-version
     strategy:
+      fail-fast: false
       matrix:
-        os: [ubuntu-latest, macos-latest, windows-latest]
+        os: [ubuntu-18.04, macos-latest, windows-latest]
         include:
-          - os: ubuntu-latest
+          - os: ubuntu-18.04
             artifact_name: meilisearch
             asset_name: meilisearch-linux-amd64
           - os: macos-latest
@@ -26,7 +51,7 @@ jobs:
     - uses: hecrj/setup-rust-action@master
       with:
         rust-version: stable
-    - uses: actions/checkout@v1
+    - uses: actions/checkout@v3
     - name: Build
       run: cargo build --release --locked
     - name: Upload binaries to release
@@ -37,50 +62,70 @@ jobs:
         asset_name: ${{ matrix.asset_name }}
         tag: ${{ github.ref }}
 
-  publish-armv7:
-    name: Publish for ARMv7
-    runs-on: ubuntu-18.04
-    steps:
-      - uses: actions/checkout@v1.0.0
-      - uses: uraimo/run-on-arch-action@v1.0.7
-        id: runcmd
-        with:
-          architecture: armv7
-          distribution: ubuntu18.04
-          run: |
-            apt update
-            apt install -y curl gcc make
-            curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain stable
-            source $HOME/.cargo/env
-            cargo build --release --locked
-      - name: Upload the binary to release
-        uses: svenstaro/upload-release-action@v1-release
-        with:
-          repo_token: ${{ secrets.PUBLISH_TOKEN }}
-          file: target/release/meilisearch
-          asset_name: meilisearch-linux-armv7
-          tag: ${{ github.ref }}
+  publish-aarch64:
+    name: Publish binary for aarch64
+    runs-on: ${{ matrix.os }}
+    needs: check-version
+    continue-on-error: false
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - build: aarch64
+            os: ubuntu-18.04
+            target: aarch64-unknown-linux-gnu
+            linker: gcc-aarch64-linux-gnu
+            use-cross: true
+            asset_name: meilisearch-linux-aarch64
 
-  publish-armv8:
-    name: Publish for ARMv8
-    runs-on: ubuntu-18.04
     steps:
-      - uses: actions/checkout@v1.0.0
-      - uses: uraimo/run-on-arch-action@v1.0.7
-        id: runcmd
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Installing Rust toolchain
+        uses: actions-rs/toolchain@v1
         with:
-          architecture: aarch64 # aka ARMv8
-          distribution: ubuntu18.04
-          run: |
-            apt update
-            apt install -y curl gcc make
-            curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain stable
-            source $HOME/.cargo/env
-            cargo build --release --locked
+          toolchain: stable
+          profile: minimal
+          target: ${{ matrix.target }}
+          override: true
+
+      - name: APT update
+        run: |
+          sudo apt update
+
+      - name: Install target specific tools
+        if: matrix.use-cross
+        run: |
+          sudo apt-get install -y ${{ matrix.linker }}
+
+      - name: Configure target aarch64 GNU
+        if: matrix.target == 'aarch64-unknown-linux-gnu'
+        ## Environment variable is not passed using env:
+        ## LD gold won't work with MUSL
+        # env:
+        #   JEMALLOC_SYS_WITH_LG_PAGE: 16
+        #   RUSTFLAGS: '-Clink-arg=-fuse-ld=gold'
+        run: |
+          echo '[target.aarch64-unknown-linux-gnu]' >> ~/.cargo/config
+          echo 'linker = "aarch64-linux-gnu-gcc"' >> ~/.cargo/config
+          echo 'JEMALLOC_SYS_WITH_LG_PAGE=16' >> $GITHUB_ENV
+          echo RUSTFLAGS="-Clink-arg=-fuse-ld=gold" >> $GITHUB_ENV
+
+      - name: Cargo build
+        uses: actions-rs/cargo@v1
+        with:
+          command: build
+          use-cross: ${{ matrix.use-cross }}
+          args: --release --target ${{ matrix.target }}
+
+      - name: List target output files
+        run: ls -lR ./target
+
       - name: Upload the binary to release
         uses: svenstaro/upload-release-action@v1-release
         with:
           repo_token: ${{ secrets.PUBLISH_TOKEN }}
-          file: target/release/meilisearch
-          asset_name: meilisearch-linux-armv8
+          file: target/${{ matrix.target }}/release/meilisearch
+          asset_name: ${{ matrix.asset_name }}
           tag: ${{ github.ref }}

.github/workflows/publish-deb-brew-pkg.yml

@@ -5,16 +5,25 @@ on:
     types: [released]
 
 jobs:
+  check-version:
+    name: Check the version validity
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Check release validity
+        run: bash .github/scripts/check-release.sh
+
   debian:
     name: Publish debian packagge
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-18.04
+    needs: check-version
     steps:
     - uses: hecrj/setup-rust-action@master
       with:
         rust-version: stable
     - name: Install cargo-deb
       run: cargo install cargo-deb
-    - uses: actions/checkout@v1
+    - uses: actions/checkout@v3
     - name: Build deb package
       run: cargo deb -p meilisearch-http -o target/debian/meilisearch.deb
     - name: Upload debian pkg to release
@@ -29,7 +38,8 @@ jobs:
 
   homebrew:
     name: Bump Homebrew formula
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-18.04
+    needs: check-version
     steps:
       - name: Create PR to Homebrew
         uses: mislav/bump-homebrew-formula-action@v1

.github/workflows/publish-docker-images.yml

@@ -0,0 +1,71 @@
+---
+on:
+  schedule:
+    - cron: '0 4 * * *' # Every day at 4:00am
+  push:
+    tags:
+      - '*'
+
+name: Publish tagged images to Docker Hub
+
+jobs:
+  docker:
+    runs-on: docker
+    steps:
+      - uses: actions/checkout@v2
+
+      # Check if the tag has the v<nmumber>.<number>.<number> format. If yes, it means we are publishing an official release.
+      # In this situation, we need to set `output.stable` to create/update the following tags (additionally to the `vX.Y.Z` Docker tag):
+      # - a `vX.Y` (without patch version) Docker tag
+      # - a `latest` Docker tag
+      - name: Check tag format
+        if: github.event_name != 'schedule'
+        id: check-tag-format
+        run: |
+          escaped_tag=$(printf "%q" ${{ github.ref_name }})
+
+          if [[ $escaped_tag =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+            echo ::set-output name=stable::true
+          else
+            echo ::set-output name=stable::false
+          fi
+
+      # Check only the validity of the tag for official releases (not for pre-releases or other tags)
+      - name: Check release validity
+        if: github.event_name != 'schedule' && steps.check-tag-format.outputs.stable == 'true'
+        run: bash .github/scripts/check-release.sh
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Login to Docker Hub
+        if: github.event_name != 'schedule'
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v4
+        with:
+          images: getmeili/meilisearch
+          # The lastest and `vX.Y` tags are only pushed for the official Meilisearch releases
+          # See https://github.com/docker/metadata-action#latest-tag
+          flavor: latest=false
+          tags: |
+            type=ref,event=tag
+            type=semver,pattern=v{{major}}.{{minor}},enable=${{ steps.check-tag-format.outputs.stable == 'true' }}
+            type=raw,value=latest,enable=${{ steps.check-tag-format.outputs.stable == 'true' }}
+
+      - name: Build and push
+        id: docker_build
+        uses: docker/build-push-action@v3
+        with:
+          # We do not push tags for the cron jobs, this is only for test purposes
+          push: ${{ github.event_name != 'schedule' }}
+          platforms: linux/amd64,linux/arm64
+          tags: ${{ steps.meta.outputs.tags }}

.github/workflows/publish-docker-latest.yml

@@ -1,22 +0,0 @@
----
-on:
-  release:
-    types: [released]
-
-name: Publish latest image to Docker Hub
-
-jobs:
-  build:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-      - name: Check if current release is latest
-        run: echo "##[set-output name=is_latest;]$(sh .github/is-latest-release.sh)"
-        id: release
-      - name: Publish to Registry
-        if: steps.release.outputs.is_latest == 'true'
-        uses: elgohr/Publish-Docker-Github-Action@master
-        with:
-          name: getmeili/meilisearch
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_PASSWORD }}

.github/workflows/publish-docker-tag.yml

@@ -1,20 +0,0 @@
----
-on:
-  push:
-    tags:
-      - '*'
-
-name: Publish tagged image to Docker Hub
-
-jobs:
-  build:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v1
-      - name: Publish to Registry
-        uses: elgohr/Publish-Docker-Github-Action@master
-        with:
-          name: getmeili/meilisearch
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_PASSWORD }}
-          tag_names: true

.github/workflows/rust.yml

@@ -0,0 +1,96 @@
+name: Rust
+
+on:
+  workflow_dispatch:
+  pull_request:
+  push:
+    # trying and staging branches are for Bors config
+    branches:
+      - trying
+      - staging
+
+env:
+  CARGO_TERM_COLOR: always
+  RUST_BACKTRACE: 1
+  RUSTFLAGS: "-D warnings"
+
+jobs:
+  tests:
+    name: Tests on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-18.04, macos-latest, windows-latest]
+    steps:
+    - uses: actions/checkout@v3
+    - uses: rui314/setup-mold@v1 # Optimize link time
+    - name: Cache dependencies
+      uses: Swatinem/rust-cache@v2.0.0
+    - name: Run cargo check without any default features
+      uses: actions-rs/cargo@v1
+      with:
+        command: build
+        args: --locked --release --no-default-features
+    - name: Run cargo test
+      uses: actions-rs/cargo@v1
+      with:
+        command: test
+        args: --locked --release
+
+  # We run tests in debug also, to make sure that the debug_assertions are hit
+  test-debug:
+    name: Run tests in debug
+    runs-on: ubuntu-18.04
+    steps:
+      - uses: actions/checkout@v3
+      - uses: rui314/setup-mold@v1 # Optimize link time
+      - uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          toolchain: stable
+          override: true
+      - name: Cache dependencies
+        uses: Swatinem/rust-cache@v2.0.0
+      - name: Run tests in debug
+        uses: actions-rs/cargo@v1
+        with:
+          command: test
+          args: --locked
+
+  clippy:
+    name: Run Clippy
+    runs-on: ubuntu-18.04
+    steps:
+      - uses: actions/checkout@v3
+      - uses: rui314/setup-mold@v1 # Optimize link time
+      - uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          toolchain: stable
+          override: true
+          components: clippy
+      - name: Cache dependencies
+        uses: Swatinem/rust-cache@v2.0.0
+      - name: Run cargo clippy
+        uses: actions-rs/cargo@v1
+        with:
+          command: clippy
+          args: --all-targets -- --deny warnings
+
+  fmt:
+    name: Run Rustfmt
+    runs-on: ubuntu-18.04
+    steps:
+      - uses: actions/checkout@v3
+      - uses: rui314/setup-mold@v1 # Optimize link time
+      - uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          toolchain: stable
+          override: true
+          components: rustfmt
+      - name: Cache dependencies
+        uses: Swatinem/rust-cache@v2.0.0
+      - name: Run cargo fmt
+        run: cargo fmt --all -- --check

.github/workflows/test.yml

@@ -1,93 +0,0 @@
----
-on:
-  push:
-    branches:
-      - release-v*
-      - trying
-      - staging
-    tags:
-      - 'v[0-9]+.[0-9]+.[0-9]+' # this only concerns tags on stable
-
-name: Test binaries with cargo test
-
-jobs:
-  check:
-    name: Test on ${{ matrix.os }}
-    runs-on: ${{ matrix.os }}
-    strategy:
-      matrix:
-        os: [ubuntu-latest, macos-latest]
-    steps:
-    - uses: actions/checkout@v1
-    - uses: actions-rs/toolchain@v1
-      with:
-        profile: minimal
-        toolchain: stable
-        override: true
-        components: clippy
-    - name: Run cargo test
-      uses: actions-rs/cargo@v1
-      with:
-        command: test
-        args: --locked --release
-    - name: Run cargo clippy
-      uses: actions-rs/cargo@v1
-      with:
-        command: clippy
-
-  build-image:
-    name: Test the build of Docker image
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v1
-    - run: docker build . --file Dockerfile -t meilisearch
-      name: Docker build
-
-  ## A push occurred on a release branch, a prerelease is created and assets are generated
-  prerelease:
-    name: create prerelease
-    needs: [check, build-image]
-    if: ${{ contains(github.ref, 'release-') && github.event_name == 'push' }}
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v2
-        with:
-          fetch-depth: 0
-      - name: Get version number
-        id: version-number
-        run: echo "##[set-output name=number;]$(echo ${{ github.ref }} | sed 's/.*\(v.*\)/\1/')"
-      - name: Get commit count
-        id: commit-count
-        run: echo "##[set-output name=count;]$(git rev-list remotes/origin/master..remotes/origin/release-${{ steps.version-number.outputs.number }} --count)"
-      - name: Create Release
-        id: create_release
-        uses: actions/create-release@v1
-        env:
-          GITHUB_TOKEN: ${{ secrets.PUBLISH_TOKEN }} # Personal Access Token
-        with:
-          tag_name: ${{ steps.version-number.outputs.number }}rc${{ steps.commit-count.outputs.count }}
-          release_name: Pre-release ${{ steps.version-number.outputs.number }}-rc${{ steps.commit-count.outputs.count }}
-          prerelease: true
-
-  ## If a tag is pushed, a release is created for this tag, and assets will be generated
-  release:
-    name: create release
-    needs: [check, build-image]
-    if:  ${{ contains(github.ref, 'tags/v') }}
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v2
-      - name: Get version number
-        id: version-number
-        run: echo "##[set-output name=number;]$(echo ${{ github.ref }} | sed 's/.*\(v.*\)/\1/')" 
-      - name: Create Release
-        id: create_release
-        uses: actions/create-release@v1
-        env:
-          GITHUB_TOKEN: ${{ secrets.PUBLISH_TOKEN }} # PAT
-        with:
-          tag_name: ${{ steps.version-number.outputs.number }}
-          release_name: Meilisearch ${{ steps.version-number.outputs.number }}
-          prerelease: false

.gitignore

@@ -1,8 +1,9 @@
 /target
-meilisearch-core/target
 **/*.csv
 **/*.json_lines
 **/*.rs.bk
 /*.mdb
 /query-history.txt
 /data.ms
+/snapshots
+/dumps

CHANGELOG.md

@@ -1,74 +0,0 @@
-## v0.14.0
-
-  - Fix facet distribution case (#797)
-  - Snapshotting (#839)
-  - Fix bucket-sort unwrap bug (#915)
-
-## v0.13.0
-
-  - placeholder search (#771)
-  - Add database version mismatch check (#794)
-  - Displayed and searchable attributes wildcard (#846)
-  - Remove sys-info route (#810)
-  - Check database version mismatch (#794)
-  - Fix unique docid bug (#841)
-  - Error codes in updates (#792)
-  - Sentry disable argument (#813)
-  - Log analytics if enabled (#825)
-  - Fix default values displayed on web interface (#874)
-
-## v0.12.0
-
-  - Fix long documents not being indexed completely bug (#816)
-  - Fix distinct attribute returning id instead of name (#800)
-  - error code rename (#805)
-
-## v0.11.1
-
-  - Fix facet cache on document update (#789) 
-  - Improvements on settings consistency (#778)
-
-## v0.11.0
-
-  - Change the HTTP framework, moving from tide to actix-web (#601)
-  - Bump sentry version to 0.18.1 (#690)
-  - Enable max payload size override (#684)
-  - Disable sentry in debug (#681)
-  - Better terminal greeting (#680)
-  - Fix highlight misalignment (#679)
-  - Add support for facet count (#676)
-  - Add support for faceted search (#631)
-  - Add support for configuring the lmdb map size (#646, #647)
-  - Add exposed port for Dockerfile (#654)
-  - Add sentry probe (#664)
-  - Fix url trailing slash and double slash issues (#659)
-  - Fix accept all Content-Type by default (#653)
-  - Return the error message from Serde when a deserialization error is encountered (#661)
-  - Fix NormalizePath middleware to make the dashboard accessible (#695)
-  - Update sentry features to remove openssl (#702)
-  - Add SSL support (#669)
-  - Rename fieldsFrequency into fieldsDistribution in stats (#719)
-  - Add support for error code reporting (#703)
-  - Allow the dashboard to query private servers (#732)
-  - Add telemetry (#720)
-  - Add post route for search (#735)
-
-## v0.10.1
-
-  - Add support for floating points in filters (#640)
-  - Add '@' character as tokenizer separator (#607)
-  - Add support for filtering on arrays of strings (#611)
-
-## v0.10.0
-
-  - Refined filtering (#592)
-  - Add the number of hits in search result (#541)
-  - Add support for aligned crop in search result (#543)
-  - Sanitize the content displayed in the web interface (#539)
-  - Add support of nested null, boolean and seq values (#571 and #568, #574)
-  - Fixed the core benchmark (#576)
-  - Publish an ARMv7 and ARMv8 binaries on releases (#540 and #581)
-  - Fixed a bug where the result of the update status after the first update was empty (#542)
-  - Fixed a bug where stop words were not handled correctly (#594)
-  - Fix CORS issues (#602)
-  - Support wildcard on attributes to retrieve, highlight, and crop (#549, #565, and #598)

CONTRIBUTING.md

@@ -0,0 +1,110 @@
+# Contributing
+
+First, thank you for contributing to Meilisearch! The goal of this document is to provide everything you need to start contributing to Meilisearch.
+
+Remember that there are many ways to contribute other than writing code: writing [tutorials or blog posts](https://github.com/meilisearch/awesome-meilisearch), improving [the documentation](https://github.com/meilisearch/documentation), submitting [bug reports](https://github.com/meilisearch/meilisearch/issues/new?assignees=&labels=&template=bug_report.md&title=) and [feature requests](https://github.com/meilisearch/product/discussions/categories/feedback-feature-proposal)...
+
+## Table of Contents
+
+- [Assumptions](#assumptions)
+- [How to Contribute](#how-to-contribute)
+- [Development Workflow](#development-workflow)
+- [Git Guidelines](#git-guidelines)
+- [Release Process (for internal team only)](#release-process-for-internal-team-only)
+
+## Assumptions
+
+1. **You're familiar with [GitHub](https://github.com) and the [Pull Requests](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests)(PR) workflow.**
+2. **You've read the Meilisearch [documentation](https://docs.meilisearch.com).**
+3. **You know about the [Meilisearch community](https://docs.meilisearch.com/learn/what_is_meilisearch/contact.html).
+   Please use this for help.**
+
+## How to Contribute
+
+1. Ensure your change has an issue! Find an
+   [existing issue](https://github.com/meilisearch/meilisearch/issues/) or [open a new issue](https://github.com/meilisearch/meilisearch/issues/new).
+   * This is where you can get a feel if the change will be accepted or not.
+2. Once approved, [fork the Meilisearch repository](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) in your own GitHub account.
+3. [Create a new Git branch](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-and-deleting-branches-within-your-repository)
+4. Review the [Development Workflow](#development-workflow) section that describes the steps to maintain the repository.
+5. Make your changes on your branch.
+6. [Submit the branch as a Pull Request](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork) pointing to the `main` branch of the Meilisearch repository. A maintainer should comment and/or review your Pull Request within a few days. Although depending on the circumstances, it may take longer.
+
+## Development Workflow
+
+### Setup and run Meilisearch
+
+```bash
+cargo run --release
+```
+
+We recommend using the `--release` flag to test the full performance of Meilisearch.
+
+### Test
+
+```bash
+cargo test
+```
+
+This command will be triggered to each PR as a requirement for merging it.
+
+If you get a "Too many open files" error you might want to increase the open file limit using this command:
+
+```bash
+ulimit -Sn 3000
+```
+
+## Git Guidelines
+
+### Git Branches
+
+All changes must be made in a branch and submitted as PR.
+
+We do not enforce any branch naming style, but please use something descriptive of your changes.
+
+### Git Commits
+
+As minimal requirements, your commit message should:
+- be capitalized
+- not finish by a dot or any other punctuation character (!,?)
+- start with a verb so that we can read your commit message this way: "This commit will ...", where "..." is the commit message.
+  e.g.: "Fix the home page button" or "Add more tests for create_index method"
+
+We don't follow any other convention, but if you want to use one, we recommend [the Chris Beams one](https://chris.beams.io/posts/git-commit/).
+
+### GitHub Pull Requests
+
+Some notes on GitHub PRs:
+
+- All PRs must be reviewed and approved by at least one maintainer.
+- The PR title should be accurate and descriptive of the changes.
+- [Convert your PR as a draft](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/changing-the-stage-of-a-pull-request) if your changes are a work in progress: no one will review it until you pass your PR as ready for review.<br>
+  The draft PRs are recommended when you want to show that you are working on something and make your work visible.
+- The branch related to the PR must be **up-to-date with `main`** before merging. Fortunately, this project uses [Bors](https://github.com/bors-ng/bors-ng) to automatically enforce this requirement without the PR author having to rebase manually.
+
+## Release Process (for internal team only)
+
+Meilisearch tools follow the [Semantic Versioning Convention](https://semver.org/).
+
+### Automation to rebase and Merge the PRs
+
+This project integrates a bot that helps us manage pull requests merging.<br>
+_[Read more about this](https://github.com/meilisearch/integration-guides/blob/main/resources/bors.md)._
+
+### How to Publish a new Release
+
+The full Meilisearch release process is described in [this guide](https://github.com/meilisearch/core-team/blob/main/resources/meilisearch-release.md). Please follow it carefully before doing any release.
+
+### Release assets
+
+For each release, the following assets are created:
+- Binaries for differents platforms (Linux, MacOS, Windows and ARM architectures) are attached to the GitHub release
+- Binaries are pushed to HomeBrew and APT (not published for RC)
+- Docker tags are created/updated:
+  - `vX.Y.Z`
+  - `vX.Y` (not published for RC)
+  - `latest` (not published for RC)
+
+<hr>
+
+Thank you again for reading this through, we can not wait to begin to work with you if you made your way through this contributing guide ❤️

Cargo.toml

@@ -1,11 +1,15 @@
 [workspace]
+resolver = "2"
 members = [
-    "meilisearch-core",
     "meilisearch-http",
-    "meilisearch-schema",
-    "meilisearch-tokenizer",
     "meilisearch-types",
+    "meilisearch-lib",
+    "meilisearch-auth",
+    "permissive-json-pointer",
 ]
 
-[profile.release]
-debug = true
+[profile.dev.package.flate2]
+opt-level = 3
+
+[profile.dev.package.milli]
+opt-level = 3

Cross.toml

@@ -0,0 +1,7 @@
+[build.env]
+passthrough = [
+    "RUST_BACKTRACE",
+    "CARGO_TERM_COLOR",
+    "RUSTFLAGS",
+    "JEMALLOC_SYS_WITH_LG_PAGE"
+]

Dockerfile

@@ -1,29 +1,46 @@
 # Compile
-FROM    alpine:3.10 AS compiler
+FROM    rust:alpine3.14 AS compiler
 
-RUN     apk update --quiet
-RUN     apk add curl
-RUN     apk add build-base
-
-RUN     curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+RUN     apk add -q --update-cache --no-cache build-base openssl-dev
 
 WORKDIR /meilisearch
 
-COPY    . .
-
+ARG     COMMIT_SHA
+ARG     COMMIT_DATE
+ENV     COMMIT_SHA=${COMMIT_SHA} COMMIT_DATE=${COMMIT_DATE}
 ENV     RUSTFLAGS="-C target-feature=-crt-static"
 
-RUN     $HOME/.cargo/bin/cargo build --release
+COPY    . .
+RUN     set -eux; \
+        apkArch="$(apk --print-arch)"; \
+        if [ "$apkArch" = "aarch64" ]; then \
+            export JEMALLOC_SYS_WITH_LG_PAGE=16; \
+        fi && \
+        cargo build --release
 
 # Run
-FROM    alpine:3.10
-
-RUN     apk add -q --no-cache libgcc tini
-
-COPY    --from=compiler /meilisearch/target/release/meilisearch .
+FROM    alpine:3.14
 
 ENV     MEILI_HTTP_ADDR 0.0.0.0:7700
+ENV     MEILI_SERVER_PROVIDER docker
+
+RUN     apk update --quiet \
+        && apk add -q --no-cache libgcc tini curl
+
+# add meilisearch to the `/bin` so you can run it from anywhere and it's easy
+# to find.
+COPY    --from=compiler /meilisearch/target/release/meilisearch /bin/meilisearch
+# To stay compatible with the older version of the container (pre v0.27.0) we're
+# going to symlink the meilisearch binary in the path to `/meilisearch`
+RUN     ln -s /bin/meilisearch /meilisearch
+
+# This directory should hold all the data related to meilisearch so we're going
+# to move our PWD in there.
+# We don't want to put the meilisearch binary
+WORKDIR /meili_data
+
+
 EXPOSE  7700/tcp
 
 ENTRYPOINT ["tini", "--"]
-CMD     ./meilisearch
+CMD     /bin/meilisearch

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2019-2020 Meili SAS
+Copyright (c) 2019-2022 Meili SAS
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,11 +1,12 @@
 <p align="center">
-  <img src="assets/logo.svg" alt="MeiliSearch" width="200" height="200" />
+  <img src="assets/logo.svg" alt="Meilisearch" width="200" height="200" />
 </p>
 
-<h1 align="center">MeiliSearch</h1>
+<h1 align="center">Meilisearch</h1>
 
 <h4 align="center">
   <a href="https://www.meilisearch.com">Website</a> |
+  <a href="https://roadmap.meilisearch.com/tabs/1-under-consideration">Roadmap</a> |
   <a href="https://blog.meilisearch.com">Blog</a> |
   <a href="https://fr.linkedin.com/company/meilisearch">LinkedIn</a> |
   <a href="https://twitter.com/meilisearch">Twitter</a> |
@@ -14,30 +15,30 @@
 </h4>
 
 <p align="center">
-  <a href="https://github.com/meilisearch/MeiliSearch/actions"><img src="https://github.com/meilisearch/MeiliSearch/workflows/Cargo%20test/badge.svg" alt="Build Status"></a>
-  <a href="https://deps.rs/repo/github/meilisearch/MeiliSearch"><img src="https://deps.rs/repo/github/meilisearch/MeiliSearch/status.svg" alt="Dependency status"></a>
-  <a href="https://github.com/meilisearch/MeiliSearch/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-MIT-informational" alt="License"></a>
-  <a href="https://slack.meilisearch.com"><img src="https://img.shields.io/badge/slack-MeiliSearch-blue.svg?logo=slack" alt="Slack"></a>
-  <a href="https://github.com/meilisearch/MeiliSearch/discussions" alt="Discussions"><img src="https://img.shields.io/badge/github-discussions-red" /></a>
+  <a href="https://github.com/meilisearch/meilisearch/actions"><img src="https://github.com/meilisearch/meilisearch/workflows/Cargo%20test/badge.svg" alt="Build Status"></a>
+  <a href="https://deps.rs/repo/github/meilisearch/meilisearch"><img src="https://deps.rs/repo/github/meilisearch/meilisearch/status.svg" alt="Dependency status"></a>
+  <a href="https://github.com/meilisearch/meilisearch/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-informational" alt="License"></a>
+  <a href="https://slack.meilisearch.com"><img src="https://img.shields.io/badge/slack-meilisearch-blue.svg?logo=slack" alt="Slack"></a>
+  <a href="https://github.com/meilisearch/meilisearch/discussions" alt="Discussions"><img src="https://img.shields.io/badge/github-discussions-red" /></a>
   <a href="https://app.bors.tech/repositories/26457"><img src="https://bors.tech/images/badge_small.svg" alt="Bors enabled"></a>
 </p>
 
 <p align="center">⚡ Lightning Fast, Ultra Relevant, and Typo-Tolerant Search Engine 🔍</p>
 
-**MeiliSearch** is a powerful, fast, open-source, easy to use and deploy search engine. Both searching and indexing are highly customizable. Features such as typo-tolerance, filters, and synonyms are provided out-of-the-box.
+**Meilisearch** is a powerful, fast, open-source, easy to use and deploy search engine. Both searching and indexing are highly customizable. Features such as typo-tolerance, filters, and synonyms are provided out-of-the-box.
 For more information about features go to [our documentation](https://docs.meilisearch.com/).
 
 <p align="center">
-  <img src="assets/movies-web-demo.gif" alt="Web interface gif" />
+  <img src="assets/trumen-fast.gif" alt="Web interface gif" />
 </p>
 
 ## ✨ Features
-* Search as-you-type experience (answers < 50 milliseconds)
+* Search-as-you-type experience (answers < 50 milliseconds)
 * Full-text search
-* Typo tolerant (understands typos and miss-spelling)
+* Typo tolerant (understands typos and misspelling)
 * Faceted search and filters
-* Supports Kanji characters
-* Supports Synonym
+* Supports hanzi (Chinese characters)
+* Supports synonyms
 * Easy to install, deploy, and maintain
 * Whole documents are returned
 * Highly customizable
@@ -47,7 +48,7 @@ For more information about features go to [our documentation](https://docs.meili
 
 ### Deploy the Server
 
-#### Brew (Mac OS)
+#### Homebrew (Mac OS)
 
 ```bash
 brew update && brew install meilisearch
@@ -57,13 +58,27 @@ meilisearch
 #### Docker
 
 ```bash
-docker run -p 7700:7700 -v $(pwd)/data.ms:/data.ms getmeili/meilisearch
+docker run -p 7700:7700 -v "$(pwd)/meili_data:/meili_data" getmeili/meilisearch
 ```
 
+#### Announcing a cloud-hosted Meilisearch
+
+Join the closed beta by filling out this [form](https://meilisearch.typeform.com/to/VI2cI2rv).
+
+#### Try Meilisearch in our Sandbox
+
+Create a Meilisearch instance in [Meilisearch Sandbox](https://sandbox.meilisearch.com/). This instance is free, and will be active for 48 hours.
+
 #### Run on Digital Ocean
 
 [![DigitalOcean Marketplace](assets/do-btn-blue.svg)](https://marketplace.digitalocean.com/apps/meilisearch?action=deploy&refcode=7c67bd97e101)
 
+#### Deploy on Platform.sh
+
+<a href="https://console.platform.sh/projects/create-project?template=https://raw.githubusercontent.com/platformsh/template-builder/master/templates/meilisearch/.platform.template.yaml&utm_content=meilisearch&utm_source=github&utm_medium=button&utm_campaign=deploy_on_platform">
+    <img src="https://platform.sh/images/deploy/lg-blue.svg" alt="Deploy on Platform.sh" width="180px" />
+</a>
+
 #### APT (Debian & Ubuntu)
 
 ```bash
@@ -84,15 +99,8 @@ curl -L https://install.meilisearch.com | sh
 If you have the latest stable Rust toolchain installed on your local system, clone the repository and change it to your working directory.
 
 ```bash
-git clone https://github.com/meilisearch/MeiliSearch.git
-cd MeiliSearch
-```
-
-In the cloned repository, compile MeiliSearch.
-
-```bash
-rustup override set stable
-rustup update stable
+git clone https://github.com/meilisearch/meilisearch.git
+cd meilisearch
 cargo run --release
 ```
 
@@ -101,17 +109,10 @@ cargo run --release
 Let's create an index! If you need a sample dataset, use [this movie database](https://www.notion.so/meilisearch/A-movies-dataset-to-test-Meili-1cbf7c9cfa4247249c40edfa22d7ca87#b5ae399b81834705ba5420ac70358a65). You can also find it in the `datasets/` directory.
 
 ```bash
-curl -L 'https://bit.ly/2PAcw9l' -o movies.json
+curl -L https://docs.meilisearch.com/movies.json -o movies.json
 ```
 
-MeiliSearch can serve multiple indexes, with different kinds of documents.
-It is required to create an index before sending documents to it.
-
-```bash
-curl -i -X POST 'http://127.0.0.1:7700/indexes' --data '{ "name": "Movies", "uid": "movies" }'
-```
-
-Now that the server knows about your brand new index, you're ready to send it some data.
+Now, you're ready to index some data.
 
 ```bash
 curl -i -X POST 'http://127.0.0.1:7700/indexes/movies/documents' \
@@ -138,55 +139,67 @@ curl 'http://127.0.0.1:7700/indexes/movies/search?q=botman+robin&limit=2' | jq
       "id": "415",
       "title": "Batman & Robin",
       "poster": "https://image.tmdb.org/t/p/w1280/79AYCcxw3kSKbhGpx1LiqaCAbwo.jpg",
-      "overview": "Along with crime-fighting partner Robin and new recruit Batgirl...",
-      "release_date": "1997-06-20",
+      "overview": "Along with crime-fighting partner Robin and new recruit Batgirl, Batman battles the dual threat of frosty genius Mr. Freeze and homicidal horticulturalist Poison Ivy. Freeze plans to put Gotham City on ice, while Ivy tries to drive a wedge between the dynamic duo.",
+      "release_date": 866768400
     },
     {
       "id": "411736",
       "title": "Batman: Return of the Caped Crusaders",
       "poster": "https://image.tmdb.org/t/p/w1280/GW3IyMW5Xgl0cgCN8wu96IlNpD.jpg",
-      "overview": "Adam West and Burt Ward returns to their iconic roles of Batman and Robin...",
-      "release_date": "2016-10-08",
+      "overview": "Adam West and Burt Ward returns to their iconic roles of Batman and Robin. Featuring the voices of Adam West, Burt Ward, and Julie Newmar, the film sees the superheroes going up against classic villains like The Joker, The Riddler, The Penguin and Catwoman, both in Gotham City… and in space.",
+      "release_date": 1475888400
     }
   ],
-  "offset": 0,
+  "nbHits": 8,
+  "exhaustiveNbHits": false,
+  "query": "botman robin",
   "limit": 2,
-  "processingTimeMs": 1,
-  "query": "botman robin"
+  "offset": 0,
+  "processingTimeMs": 2
 }
 ```
 
 #### Use the Web Interface
 
-We also deliver an **out-of-the-box web interface** in which you can test MeiliSearch interactively.
+We also deliver an **out-of-the-box [web interface](https://github.com/meilisearch/mini-dashboard)** in which you can test Meilisearch interactively.
 
-You can access the web interface in your web browser at the root of the server. The default URL is [http://127.0.0.1:7700](http://127.0.0.1:7700). All you need to do is open your web browser and enter MeiliSearch’s address to visit it. This will lead you to a web page with a search bar that will allow you to search in the selected index.
+You can access the web interface in your web browser at the root of the server. The default URL is [http://127.0.0.1:7700](http://127.0.0.1:7700). All you need to do is open your web browser and enter Meilisearch’s address to visit it. This will lead you to a web page with a search bar that will allow you to search in the selected index.
 
 | [See the gif above](#demo)
 
 ## Documentation
 
-Now that your MeiliSearch server is up and running, you can learn more about how to tune your search engine in [the documentation](https://docs.meilisearch.com).
+Now that your Meilisearch server is up and running, you can learn more about how to tune your search engine in [the documentation](https://docs.meilisearch.com).
 
 ## Contributing
 
-Hey! We're glad you're thinking about contributing to MeiliSearch! If you think something is missing or could be improved, please open issues and pull requests. If you'd like to help this project grow, we'd love to have you! To start contributing, checking [issues tagged as "good-first-issue"](https://github.com/meilisearch/MeiliSearch/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) is a good start!
+Hey! We're glad you're thinking about contributing to Meilisearch! Feel free to pick an [issue labeled as `good first issue`](https://github.com/meilisearch/meilisearch/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22), and to ask any question you need. Some points might not be clear and we are available to help you!
 
+Also, we recommend following the [CONTRIBUTING](./CONTRIBUTING.md) to create your PR.
+
+## Core engine and tokenizer
+
+The code in this repository is only concerned with managing multiple indexes, handling the update store, and exposing an HTTP API.
+
+Search and indexation are the domain of our core engine, [`milli`](https://github.com/meilisearch/milli), while tokenization is handled by [our `tokenizer` library](https://github.com/meilisearch/tokenizer/).
 ## Telemetry
 
-MeiliSearch collects anonymous data regarding general usage.
-This helps us better understand developers usage of MeiliSearch features.<br/>
-To see what information we're retrieving, please see the complete list [on the dedicated issue](https://github.com/meilisearch/MeiliSearch/issues/720).<br/>
-We also use Sentry to make us crash and error reports. If you want to know more about what Sentry collects, please visit their [privacy policy website](https://sentry.io/privacy/).<br/>
-This program is optionnal, you can disable these analytics by using the `MEILI_NO_ANALYTICS` env variable.
+Meilisearch collects anonymous data regarding general usage.
+This helps us better understand developers' usage of Meilisearch features.
+
+To find out more on what information we're retrieving, please see our documentation on [Telemetry](https://docs.meilisearch.com/learn/what_is_meilisearch/telemetry.html).
+
+This program is optional, you can disable these analytics by using the `MEILI_NO_ANALYTICS` env variable.
+
+## Feature request
+
+The feature requests are not managed in this repository. Please visit our [dedicated repository](https://github.com/meilisearch/product) to see our work about the Meilisearch product.
+
+If you have a feature request or any feedback about an existing feature, please open [a discussion](https://github.com/meilisearch/product/discussions).
+Also, feel free to participate in the current discussions, we are looking forward to reading your comments.
 
 ## 💌 Contact
 
-Feel free to contact us about any questions you may have:
-* At [bonjour@meilisearch.com](mailto:bonjour@meilisearch.com)
-* Via the chat box available on every page of [our documentation](https://docs.meilisearch.com/) and on [our landing page](https://www.meilisearch.com/).
-* 🆕 Join our [GitHub Discussions forum](https://github.com/meilisearch/MeiliSearch/discussions)
-* Join our [Slack community](https://slack.meilisearch.com/).
-* By opening an issue.
+Please visit [this page](https://docs.meilisearch.com/learn/what_is_meilisearch/contact.html#contact-us).
 
-MeiliSearch is developed by [Meili](https://www.meilisearch.com), a young company. To know more about us, you can [read our blog](https://blog.meilisearch.com). Any suggestion or feedback is highly appreciated. Thank you for your support!
+Meilisearch is developed by [Meili](https://www.meilisearch.com), a young company. To know more about us, you can [read our blog](https://blog.meilisearch.com). Any suggestion or feedback is highly appreciated. Thank you for your support!

SECURITY.md

@@ -0,0 +1,33 @@
+# Security
+
+Meilisearch takes the security of our software products and services seriously.
+
+If you believe you have found a security vulnerability in any Meilisearch-owned repository, please report it to us as described below.
+
+## Supported versions
+
+As long as we are pre-v1.0, only the latest version of Meilisearch will be supported with security updates.
+
+## Reporting security issues
+
+⚠️ Please do not report security vulnerabilities through public GitHub issues. ⚠️
+
+Instead, please kindly email us at security@meilisearch.com
+
+Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
+
+- Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
+- Full paths of source file(s) related to the manifestation of the issue
+- The location of the affected source code (tag/branch/commit or direct URL)
+- Any special configuration required to reproduce the issue
+- Step-by-step instructions to reproduce the issue
+- Proof-of-concept or exploit code (if possible)
+- Impact of the issue, including how an attacker might exploit the issue
+
+This information will help us triage your report more quickly.
+
+You will receive a response from us within 72 hours. If the issue is confirmed, we will release a patch as soon as possible depending on complexity.
+
+## Preferred languages
+
+We prefer all communications to be in English.

assets/logo.svg

@@ -1,17 +1,19 @@
-<svg width="360" height="360" viewBox="0 0 360 360" fill="none" xmlns="http://www.w3.org/2000/svg">
-<g id="logo_main">
-<rect id="Rectangle" x="107.333" y="0.150146" width="274.315" height="274.315" rx="98.8334" transform="rotate(23 107.333 0.150146)" fill="url(#paint0_linear)"/>
-<path id="Rectangle_2" fill-rule="evenodd" clip-rule="evenodd" d="M61.3296 230.199C46.2224 194.608 38.6688 176.813 38.208 160.329C37.5286 136.025 47.0175 112.539 64.3891 95.5282C76.1718 83.9904 93.9669 76.4368 129.557 61.3296C165.147 46.2224 182.943 38.6688 199.427 38.208C223.731 37.5286 247.217 47.0175 264.228 64.3891C275.766 76.1718 283.319 93.9669 298.426 129.557C313.534 165.147 321.087 182.943 321.548 199.427C322.227 223.731 312.738 247.217 295.367 264.228C283.584 275.766 265.789 283.319 230.199 298.426C194.608 313.534 176.813 321.087 160.329 321.548C136.025 322.227 112.539 312.738 95.5282 295.367C83.9903 283.584 76.4368 265.789 61.3296 230.199Z" fill="url(#paint1_linear)"/>
-<path id="m" fill-rule="evenodd" clip-rule="evenodd" d="M219.568 130.748C242.363 130.748 259.263 147.451 259.263 174.569V229.001H227.232V179.678C227.232 166.119 220.747 159.634 210.136 159.634C205.223 159.634 200.311 161.796 195.595 167.494C195.791 169.852 195.988 172.21 195.988 174.569V229.001H164.154V179.678C164.154 166.119 157.472 159.634 147.057 159.634C142.145 159.634 137.429 161.992 132.712 168.084V229.001H100.878V133.695H132.712V139.394C139.197 133.892 145.878 130.748 156.49 130.748C168.477 130.748 178.695 135.267 185.769 143.52C195.791 134.678 205.42 130.748 219.568 130.748Z" fill="white"/>
-</g>
+<svg width="300" height="300" viewBox="0 0 300 300" fill="none" xmlns="http://www.w3.org/2000/svg">
+<path d="M0 237L55.426 96.7678C63.2367 77.0063 82.499 64 103.955 64H137.371L81.9447 204.232C74.1341 223.993 54.8717 237 33.4156 237H0Z" fill="url(#paint0_linear_1_898)"/>
+<path d="M81.3123 237L136.738 96.7682C144.549 77.0067 163.811 64.0004 185.267 64.0004H218.683L163.257 204.232C155.446 223.994 136.184 237 114.728 237H81.3123Z" fill="url(#paint1_linear_1_898)"/>
+<path d="M162.629 237L218.055 96.7682C225.866 77.0067 245.128 64.0004 266.584 64.0004H300L244.574 204.232C236.763 223.994 217.501 237 196.045 237H162.629Z" fill="url(#paint2_linear_1_898)"/>
 <defs>
-<linearGradient id="paint0_linear" x1="-13.6248" y1="129.208" x2="244.49" y2="403.522" gradientUnits="userSpaceOnUse">
-<stop stop-color="#E41359"/>
-<stop offset="1" stop-color="#F23C79"/>
+<linearGradient id="paint0_linear_1_898" x1="300.001" y1="50.7858" x2="1.63474" y2="221.244" gradientUnits="userSpaceOnUse">
+<stop stop-color="#FF5CAA"/>
+<stop offset="1" stop-color="#FF4E62"/>
 </linearGradient>
-<linearGradient id="paint1_linear" x1="11.0088" y1="111.65" x2="111.65" y2="348.747" gradientUnits="userSpaceOnUse">
-<stop stop-color="#24222F"/>
-<stop offset="1" stop-color="#2B2937"/>
+<linearGradient id="paint1_linear_1_898" x1="300.001" y1="50.7858" x2="1.63474" y2="221.244" gradientUnits="userSpaceOnUse">
+<stop stop-color="#FF5CAA"/>
+<stop offset="1" stop-color="#FF4E62"/>
+</linearGradient>
+<linearGradient id="paint2_linear_1_898" x1="300.001" y1="50.7858" x2="1.63474" y2="221.244" gradientUnits="userSpaceOnUse">
+<stop stop-color="#FF5CAA"/>
+<stop offset="1" stop-color="#FF4E62"/>
 </linearGradient>
 </defs>
 </svg>

bors.toml

@@ -1,3 +1,11 @@
-status = ["Test on macos-latest", "Test on ubuntu-latest"]
-# 4 hours timeout
-timeout-sec = 14400
+status = [
+    'Tests on ubuntu-18.04',
+    'Tests on macos-latest',
+    'Tests on windows-latest',
+    'Run Clippy',
+    'Run Rustfmt',
+    'Run tests in debug',
+]
+pr_status = ['Milestone Check']
+# 3 hours timeout
+timeout-sec = 10800

bump.sh

@@ -1,38 +0,0 @@
-#!/usr/bin/bash
-
-NEW_VERSION=$1
-
-if [ -z "$NEW_VERSION" ]
-then
-	echo "error: a version number must be provided"
-	exit 1
-fi
-
-# find current version
-CURRENT_VERSION=$(cat **/*.toml | grep meilisearch | grep version | sed 's/.*\([0-9]\+\.[0-9]\+\.[0-9]\+\).*/\1/' | sed "1q;d")
-
-# bump all version in .toml
-echo "bumping from version $CURRENT_VERSION to version $NEW_VERSION"
-while true
-do
-	read -r -p "Continue (y/n)?" choice
-	case "$choice" in
-		y|Y ) break;;
-		n|N ) echo "aborting bump" && exit 0;;
-		* ) echo "invalid choice";;
-	esac
-done
-# update all crate version
-sed -i "s/version = \"$CURRENT_VERSION\"/version = \"$NEW_VERSION\"/" **/*.toml
-
-printf "running cargo check: "
-
-CARGO_CHECK=$(cargo check 2>&1)
-
-if [ $? != "0" ]
-then
-	printf "\033[31;1m FAIL \033[0m\n"
-	printf "$CARGO_CHECK"
-	exit 1
-fi
-printf "\033[32;1m OK \033[0m\n"

datasets/movies/README.md

@@ -1 +0,0 @@
-_datas in movies.csv are from https://www.themoviedb.org/_

datasets/movies/settings.json

@@ -1,10 +0,0 @@
-{
-    "searchableAttributes": ["title", "overview"],
-    "displayedAttributes": [
-        "id",
-        "title",
-        "overview",
-        "release_date",
-        "poster"
-    ]
-}

download-latest.sh

@@ -6,7 +6,6 @@ GREEN='\033[32m'
 DEFAULT='\033[0m'
 
 # GLOBALS
-BINARY_NAME='meilisearch'
 GREP_SEMVER_REGEXP='v\([0-9]*\)[.]\([0-9]*\)[.]\([0-9]*\)$' # i.e. v[number].[number].[number]
 
 # FUNCTIONS
@@ -22,7 +21,7 @@ semverParseInto() {
     eval $2=`echo $1 | sed -e "s#$RE#\1#"`
     #MINOR
     eval $3=`echo $1 | sed -e "s#$RE#\2#"`
-    #MINOR
+    #PATCH
     eval $4=`echo $1 | sed -e "s#$RE#\3#"`
     #SPECIAL
     eval $5=`echo $1 | sed -e "s#$RE#\4#"`
@@ -52,13 +51,13 @@ semverLT() {
     if [ $MAJOR_A -le $MAJOR_B ] && [ $MINOR_A -le $MINOR_B ] && [ $PATCH_A -lt $PATCH_B ]; then
         return 0
     fi
-    if [ "_$SPECIAL_A"  == "_" ] && [ "_$SPECIAL_B"  == "_" ] ; then
+    if [ "_$SPECIAL_A"  == '_' ] && [ "_$SPECIAL_B"  == '_' ] ; then
         return 1
     fi
-    if [ "_$SPECIAL_A"  == "_" ] && [ "_$SPECIAL_B"  != "_" ] ; then
+    if [ "_$SPECIAL_A"  == '_' ] && [ "_$SPECIAL_B"  != '_' ] ; then
         return 1
     fi
-    if [ "_$SPECIAL_A"  != "_" ] && [ "_$SPECIAL_B"  == "_" ] ; then
+    if [ "_$SPECIAL_A"  != '_' ] && [ "_$SPECIAL_B"  == '_' ] ; then
         return 0
     fi
     if [ "_$SPECIAL_A" < "_$SPECIAL_B" ]; then
@@ -68,39 +67,47 @@ semverLT() {
     return 1
 }
 
+# Get a token from https://github.com/settings/tokens to increase rate limit (from 60 to 5000), make sure the token scope is set to 'public_repo'
+# Create GITHUB_PAT environment variable once you acquired the token to start using it
 # Returns the tag of the latest stable release (in terms of semver and not of release date)
 get_latest() {
     temp_file='temp_file' # temp_file needed because the grep would start before the download is over
-    curl -s 'https://api.github.com/repos/meilisearch/MeiliSearch/releases' > "$temp_file"
+
+    if [ -z "$GITHUB_PAT" ]; then
+        curl -s 'https://api.github.com/repos/meilisearch/meilisearch/releases' > "$temp_file" || return 1
+    else
+        curl -H "Authorization: token $GITHUB_PAT" -s 'https://api.github.com/repos/meilisearch/meilisearch/releases' > "$temp_file" || return 1
+    fi
+
     releases=$(cat "$temp_file" | \
-        grep -E "tag_name|draft|prerelease" \
+        grep -E '"tag_name":|"draft":|"prerelease":' \
         | tr -d ',"' | cut -d ':' -f2 | tr -d ' ')
         # Returns a list of [tag_name draft_boolean prerelease_boolean ...]
         # Ex: v0.10.1 false false v0.9.1-rc.1 false true v0.9.0 false false...
 
     i=0
-    latest=""
-    current_tag=""
+    latest=''
+    current_tag=''
     for release_info in $releases; do
-        if [ $i -eq 0 ]; then # Cheking tag_name
+        if [ $i -eq 0 ]; then # Checking tag_name
             if echo "$release_info" | grep -q "$GREP_SEMVER_REGEXP"; then # If it's not an alpha or beta release
                 current_tag=$release_info
             else
-                current_tag=""
+                current_tag=''
             fi
             i=1
         elif [ $i -eq 1 ]; then # Checking draft boolean
-            if [ "$release_info" = "true" ]; then
-                current_tag=""
+            if [ "$release_info" = 'true' ]; then
+                current_tag=''
             fi
             i=2
         elif [ $i -eq 2 ]; then # Checking prerelease boolean
-            if [ "$release_info" = "true" ]; then
-                current_tag=""
+            if [ "$release_info" = 'true' ]; then
+                current_tag=''
             fi
             i=0
-            if [ "$current_tag" != "" ]; then # If the current_tag is valid
-                if [ "$latest" = "" ]; then # If there is no latest yet
+            if [ "$current_tag" != '' ]; then # If the current_tag is valid
+                if [ "$latest" = '' ]; then # If there is no latest yet
                     latest="$current_tag"
                 else
                     semverLT $current_tag $latest # Comparing latest and the current tag
@@ -113,7 +120,7 @@ get_latest() {
     done
 
     rm -f "$temp_file"
-    echo $latest
+    return 0
 }
 
 # Gets the OS by setting the $os variable
@@ -127,6 +134,9 @@ get_os() {
     'Linux')
         os='linux'
         ;;
+	'MINGW'*)
+        os='windows'
+        ;;
     *)
         return 1
     esac
@@ -138,11 +148,18 @@ get_os() {
 get_archi() {
     architecture=$(uname -m)
     case "$architecture" in
-    'x86_64' | 'amd64')
+    'x86_64' | 'amd64' )
         archi='amd64'
         ;;
+    'arm64')
+        if [ $os = 'macos' ]; then # MacOS M1
+            archi='amd64'
+        else
+            archi='aarch64'
+        fi
+        ;;
     'aarch64')
-        archi='armv8'
+        archi='aarch64'
         ;;
     *)
         return 1
@@ -151,7 +168,7 @@ get_archi() {
 }
 
 success_usage() {
-    printf "$GREEN%s\n$DEFAULT" "MeiliSearch binary successfully downloaded as '$BINARY_NAME' file."
+    printf "$GREEN%s\n$DEFAULT" "Meilisearch $latest binary successfully downloaded as '$binary_name' file."
     echo ''
     echo 'Run it:'
     echo '    $ ./meilisearch'
@@ -159,29 +176,65 @@ success_usage() {
     echo '    $ ./meilisearch --help'
 }
 
-failure_usage() {
-    printf "$RED%s\n$DEFAULT" 'ERROR: MeiliSearch binary is not available for your OS distribution or your architecture yet.'
+not_available_failure_usage() {
+    printf "$RED%s\n$DEFAULT" 'ERROR: Meilisearch binary is not available for your OS distribution or your architecture yet.'
     echo ''
     echo 'However, you can easily compile the binary from the source files.'
-    echo 'Follow the steps at the page ("Source" tab): https://docs.meilisearch.com/guides/advanced_guides/installation.html'
+    echo 'Follow the steps at the page ("Source" tab): https://docs.meilisearch.com/learn/getting_started/installation.html'
+}
+
+fetch_release_failure_usage() {
+    echo ''
+    printf "$RED%s\n$DEFAULT" 'ERROR: Impossible to get the latest stable version of Meilisearch.'
+    echo 'Please let us know about this issue: https://github.com/meilisearch/meilisearch/issues/new/choose'
 }
 
 # MAIN
-latest="$(get_latest)"
-get_os
-if [ "$?" -eq 1 ]; then
-    failure_usage
+
+# Fill $latest variable
+if ! get_latest; then
+    fetch_release_failure_usage # TO CHANGE
     exit 1
 fi
-get_archi
-if [ "$?" -eq 1 ]; then
-    failure_usage
+
+if [ "$latest" = '' ]; then
+    fetch_release_failure_usage
     exit 1
 fi
-echo "Downloading MeiliSearch binary $latest for $os, architecture $archi..."
-release_file="meilisearch-$os-$archi"
-link="https://github.com/meilisearch/MeiliSearch/releases/download/$latest/$release_file"
-curl -OL "$link"
-mv "$release_file" "$BINARY_NAME"
-chmod 744 "$BINARY_NAME"
+
+# Fill $os variable
+if ! get_os; then
+    not_available_failure_usage
+    exit 1
+fi
+
+# Fill $archi variable
+if ! get_archi; then
+    not_available_failure_usage
+    exit 1
+fi
+
+echo "Downloading Meilisearch binary $latest for $os, architecture $archi..."
+case "$os" in
+    'windows')
+        release_file="meilisearch-$os-$archi.exe"
+        binary_name='meilisearch.exe'
+
+        ;;
+    *)
+        release_file="meilisearch-$os-$archi"
+        binary_name='meilisearch'
+
+esac
+
+# Fetch the Meilisearch binary
+link="https://github.com/meilisearch/meilisearch/releases/download/$latest/$release_file"
+curl --fail -OL "$link"
+if [ $? -ne 0 ]; then
+    fetch_release_failure_usage
+    exit 1
+fi
+
+mv "$release_file" "$binary_name"
+chmod 744 "$binary_name"
 success_usage

meilisearch-auth/Cargo.toml

@@ -0,0 +1,17 @@
+[package]
+name = "meilisearch-auth"
+version = "0.28.1"
+edition = "2021"
+
+[dependencies]
+enum-iterator = "0.7.0"
+hmac = "0.12.1"
+meilisearch-types = { path = "../meilisearch-types" }
+milli = { git = "https://github.com/meilisearch/milli.git", tag = "v0.31.2" }
+rand = "0.8.4"
+serde = { version = "1.0.136", features = ["derive"] }
+serde_json = { version = "1.0.79", features = ["preserve_order"] }
+sha2 = "0.10.2"
+thiserror = "1.0.30"
+time = { version = "0.3.7", features = ["serde-well-known", "formatting", "parsing", "macros"] }
+uuid = { version = "1.1.2", features = ["serde", "v4"] }

meilisearch-auth/src/action.rs

@@ -0,0 +1,151 @@
+use enum_iterator::IntoEnumIterator;
+use serde::{Deserialize, Serialize};
+use std::hash::Hash;
+
+#[derive(IntoEnumIterator, Copy, Clone, Serialize, Deserialize, Debug, Eq, PartialEq, Hash)]
+#[repr(u8)]
+pub enum Action {
+    #[serde(rename = "*")]
+    All = actions::ALL,
+    #[serde(rename = "search")]
+    Search = actions::SEARCH,
+    #[serde(rename = "documents.*")]
+    DocumentsAll = actions::DOCUMENTS_ALL,
+    #[serde(rename = "documents.add")]
+    DocumentsAdd = actions::DOCUMENTS_ADD,
+    #[serde(rename = "documents.get")]
+    DocumentsGet = actions::DOCUMENTS_GET,
+    #[serde(rename = "documents.delete")]
+    DocumentsDelete = actions::DOCUMENTS_DELETE,
+    #[serde(rename = "indexes.*")]
+    IndexesAll = actions::INDEXES_ALL,
+    #[serde(rename = "indexes.create")]
+    IndexesAdd = actions::INDEXES_CREATE,
+    #[serde(rename = "indexes.get")]
+    IndexesGet = actions::INDEXES_GET,
+    #[serde(rename = "indexes.update")]
+    IndexesUpdate = actions::INDEXES_UPDATE,
+    #[serde(rename = "indexes.delete")]
+    IndexesDelete = actions::INDEXES_DELETE,
+    #[serde(rename = "tasks.*")]
+    TasksAll = actions::TASKS_ALL,
+    #[serde(rename = "tasks.get")]
+    TasksGet = actions::TASKS_GET,
+    #[serde(rename = "settings.*")]
+    SettingsAll = actions::SETTINGS_ALL,
+    #[serde(rename = "settings.get")]
+    SettingsGet = actions::SETTINGS_GET,
+    #[serde(rename = "settings.update")]
+    SettingsUpdate = actions::SETTINGS_UPDATE,
+    #[serde(rename = "stats.*")]
+    StatsAll = actions::STATS_ALL,
+    #[serde(rename = "stats.get")]
+    StatsGet = actions::STATS_GET,
+    #[serde(rename = "dumps.*")]
+    DumpsAll = actions::DUMPS_ALL,
+    #[serde(rename = "dumps.create")]
+    DumpsCreate = actions::DUMPS_CREATE,
+    #[serde(rename = "version")]
+    Version = actions::VERSION,
+    #[serde(rename = "keys.create")]
+    KeysAdd = actions::KEYS_CREATE,
+    #[serde(rename = "keys.get")]
+    KeysGet = actions::KEYS_GET,
+    #[serde(rename = "keys.update")]
+    KeysUpdate = actions::KEYS_UPDATE,
+    #[serde(rename = "keys.delete")]
+    KeysDelete = actions::KEYS_DELETE,
+}
+
+impl Action {
+    pub fn from_repr(repr: u8) -> Option<Self> {
+        use actions::*;
+        match repr {
+            ALL => Some(Self::All),
+            SEARCH => Some(Self::Search),
+            DOCUMENTS_ALL => Some(Self::DocumentsAll),
+            DOCUMENTS_ADD => Some(Self::DocumentsAdd),
+            DOCUMENTS_GET => Some(Self::DocumentsGet),
+            DOCUMENTS_DELETE => Some(Self::DocumentsDelete),
+            INDEXES_ALL => Some(Self::IndexesAll),
+            INDEXES_CREATE => Some(Self::IndexesAdd),
+            INDEXES_GET => Some(Self::IndexesGet),
+            INDEXES_UPDATE => Some(Self::IndexesUpdate),
+            INDEXES_DELETE => Some(Self::IndexesDelete),
+            TASKS_ALL => Some(Self::TasksAll),
+            TASKS_GET => Some(Self::TasksGet),
+            SETTINGS_ALL => Some(Self::SettingsAll),
+            SETTINGS_GET => Some(Self::SettingsGet),
+            SETTINGS_UPDATE => Some(Self::SettingsUpdate),
+            STATS_ALL => Some(Self::StatsAll),
+            STATS_GET => Some(Self::StatsGet),
+            DUMPS_ALL => Some(Self::DumpsAll),
+            DUMPS_CREATE => Some(Self::DumpsCreate),
+            VERSION => Some(Self::Version),
+            KEYS_CREATE => Some(Self::KeysAdd),
+            KEYS_GET => Some(Self::KeysGet),
+            KEYS_UPDATE => Some(Self::KeysUpdate),
+            KEYS_DELETE => Some(Self::KeysDelete),
+            _otherwise => None,
+        }
+    }
+
+    pub fn repr(&self) -> u8 {
+        use actions::*;
+        match self {
+            Self::All => ALL,
+            Self::Search => SEARCH,
+            Self::DocumentsAll => DOCUMENTS_ALL,
+            Self::DocumentsAdd => DOCUMENTS_ADD,
+            Self::DocumentsGet => DOCUMENTS_GET,
+            Self::DocumentsDelete => DOCUMENTS_DELETE,
+            Self::IndexesAll => INDEXES_ALL,
+            Self::IndexesAdd => INDEXES_CREATE,
+            Self::IndexesGet => INDEXES_GET,
+            Self::IndexesUpdate => INDEXES_UPDATE,
+            Self::IndexesDelete => INDEXES_DELETE,
+            Self::TasksAll => TASKS_ALL,
+            Self::TasksGet => TASKS_GET,
+            Self::SettingsAll => SETTINGS_ALL,
+            Self::SettingsGet => SETTINGS_GET,
+            Self::SettingsUpdate => SETTINGS_UPDATE,
+            Self::StatsAll => STATS_ALL,
+            Self::StatsGet => STATS_GET,
+            Self::DumpsAll => DUMPS_ALL,
+            Self::DumpsCreate => DUMPS_CREATE,
+            Self::Version => VERSION,
+            Self::KeysAdd => KEYS_CREATE,
+            Self::KeysGet => KEYS_GET,
+            Self::KeysUpdate => KEYS_UPDATE,
+            Self::KeysDelete => KEYS_DELETE,
+        }
+    }
+}
+
+pub mod actions {
+    pub(crate) const ALL: u8 = 0;
+    pub const SEARCH: u8 = 1;
+    pub const DOCUMENTS_ALL: u8 = 2;
+    pub const DOCUMENTS_ADD: u8 = 3;
+    pub const DOCUMENTS_GET: u8 = 4;
+    pub const DOCUMENTS_DELETE: u8 = 5;
+    pub const INDEXES_ALL: u8 = 6;
+    pub const INDEXES_CREATE: u8 = 7;
+    pub const INDEXES_GET: u8 = 8;
+    pub const INDEXES_UPDATE: u8 = 9;
+    pub const INDEXES_DELETE: u8 = 10;
+    pub const TASKS_ALL: u8 = 11;
+    pub const TASKS_GET: u8 = 12;
+    pub const SETTINGS_ALL: u8 = 13;
+    pub const SETTINGS_GET: u8 = 14;
+    pub const SETTINGS_UPDATE: u8 = 15;
+    pub const STATS_ALL: u8 = 16;
+    pub const STATS_GET: u8 = 17;
+    pub const DUMPS_ALL: u8 = 18;
+    pub const DUMPS_CREATE: u8 = 19;
+    pub const VERSION: u8 = 20;
+    pub const KEYS_CREATE: u8 = 21;
+    pub const KEYS_GET: u8 = 22;
+    pub const KEYS_UPDATE: u8 = 23;
+    pub const KEYS_DELETE: u8 = 24;
+}

meilisearch-auth/src/dump.rs

@@ -0,0 +1,47 @@
+use serde_json::Deserializer;
+
+use std::fs::File;
+use std::io::BufReader;
+use std::io::Write;
+use std::path::Path;
+
+use crate::{AuthController, HeedAuthStore, Result};
+
+const KEYS_PATH: &str = "keys";
+
+impl AuthController {
+    pub fn dump(src: impl AsRef<Path>, dst: impl AsRef<Path>) -> Result<()> {
+        let mut store = HeedAuthStore::new(&src)?;
+
+        // do not attempt to close the database on drop!
+        store.set_drop_on_close(false);
+
+        let keys_file_path = dst.as_ref().join(KEYS_PATH);
+
+        let keys = store.list_api_keys()?;
+        let mut keys_file = File::create(&keys_file_path)?;
+        for key in keys {
+            serde_json::to_writer(&mut keys_file, &key)?;
+            keys_file.write_all(b"\n")?;
+        }
+
+        Ok(())
+    }
+
+    pub fn load_dump(src: impl AsRef<Path>, dst: impl AsRef<Path>) -> Result<()> {
+        let store = HeedAuthStore::new(&dst)?;
+
+        let keys_file_path = src.as_ref().join(KEYS_PATH);
+
+        if !keys_file_path.exists() {
+            return Ok(());
+        }
+
+        let reader = BufReader::new(File::open(&keys_file_path)?);
+        for key in Deserializer::from_reader(reader).into_iter() {
+            store.put_api_key(key?)?;
+        }
+
+        Ok(())
+    }
+}

meilisearch-auth/src/error.rs

@@ -0,0 +1,60 @@
+use std::error::Error;
+
+use meilisearch_types::error::{Code, ErrorCode};
+use meilisearch_types::internal_error;
+use serde_json::Value;
+
+pub type Result<T> = std::result::Result<T, AuthControllerError>;
+
+#[derive(Debug, thiserror::Error)]
+pub enum AuthControllerError {
+    #[error("`{0}` field is mandatory.")]
+    MissingParameter(&'static str),
+    #[error("`actions` field value `{0}` is invalid. It should be an array of string representing action names.")]
+    InvalidApiKeyActions(Value),
+    #[error("`indexes` field value `{0}` is invalid. It should be an array of string representing index names.")]
+    InvalidApiKeyIndexes(Value),
+    #[error("`expiresAt` field value `{0}` is invalid. It should follow the RFC 3339 format to represents a date or datetime in the future or specified as a null value. e.g. 'YYYY-MM-DD' or 'YYYY-MM-DD HH:MM:SS'.")]
+    InvalidApiKeyExpiresAt(Value),
+    #[error("`description` field value `{0}` is invalid. It should be a string or specified as a null value.")]
+    InvalidApiKeyDescription(Value),
+    #[error(
+        "`name` field value `{0}` is invalid. It should be a string or specified as a null value."
+    )]
+    InvalidApiKeyName(Value),
+    #[error("`uid` field value `{0}` is invalid. It should be a valid UUID v4 string or omitted.")]
+    InvalidApiKeyUid(Value),
+    #[error("API key `{0}` not found.")]
+    ApiKeyNotFound(String),
+    #[error("`uid` field value `{0}` is already an existing API key.")]
+    ApiKeyAlreadyExists(String),
+    #[error("The `{0}` field cannot be modified for the given resource.")]
+    ImmutableField(String),
+    #[error("Internal error: {0}")]
+    Internal(Box<dyn Error + Send + Sync + 'static>),
+}
+
+internal_error!(
+    AuthControllerError: milli::heed::Error,
+    std::io::Error,
+    serde_json::Error,
+    std::str::Utf8Error
+);
+
+impl ErrorCode for AuthControllerError {
+    fn error_code(&self) -> Code {
+        match self {
+            Self::MissingParameter(_) => Code::MissingParameter,
+            Self::InvalidApiKeyActions(_) => Code::InvalidApiKeyActions,
+            Self::InvalidApiKeyIndexes(_) => Code::InvalidApiKeyIndexes,
+            Self::InvalidApiKeyExpiresAt(_) => Code::InvalidApiKeyExpiresAt,
+            Self::InvalidApiKeyDescription(_) => Code::InvalidApiKeyDescription,
+            Self::InvalidApiKeyName(_) => Code::InvalidApiKeyName,
+            Self::ApiKeyNotFound(_) => Code::ApiKeyNotFound,
+            Self::InvalidApiKeyUid(_) => Code::InvalidApiKeyUid,
+            Self::ApiKeyAlreadyExists(_) => Code::ApiKeyAlreadyExists,
+            Self::ImmutableField(_) => Code::ImmutableField,
+            Self::Internal(_) => Code::Internal,
+        }
+    }
+}

meilisearch-auth/src/key.rs

@@ -0,0 +1,201 @@
+use crate::action::Action;
+use crate::error::{AuthControllerError, Result};
+use crate::store::KeyId;
+
+use meilisearch_types::index_uid::IndexUid;
+use meilisearch_types::star_or::StarOr;
+use serde::{Deserialize, Serialize};
+use serde_json::{from_value, Value};
+use time::format_description::well_known::Rfc3339;
+use time::macros::{format_description, time};
+use time::{Date, OffsetDateTime, PrimitiveDateTime};
+use uuid::Uuid;
+
+#[derive(Debug, Deserialize, Serialize)]
+pub struct Key {
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub description: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub name: Option<String>,
+    pub uid: KeyId,
+    pub actions: Vec<Action>,
+    pub indexes: Vec<StarOr<IndexUid>>,
+    #[serde(with = "time::serde::rfc3339::option")]
+    pub expires_at: Option<OffsetDateTime>,
+    #[serde(with = "time::serde::rfc3339")]
+    pub created_at: OffsetDateTime,
+    #[serde(with = "time::serde::rfc3339")]
+    pub updated_at: OffsetDateTime,
+}
+
+impl Key {
+    pub fn create_from_value(value: Value) -> Result<Self> {
+        let name = match value.get("name") {
+            None | Some(Value::Null) => None,
+            Some(des) => from_value(des.clone())
+                .map(Some)
+                .map_err(|_| AuthControllerError::InvalidApiKeyName(des.clone()))?,
+        };
+
+        let description = match value.get("description") {
+            None | Some(Value::Null) => None,
+            Some(des) => from_value(des.clone())
+                .map(Some)
+                .map_err(|_| AuthControllerError::InvalidApiKeyDescription(des.clone()))?,
+        };
+
+        let uid = value.get("uid").map_or_else(
+            || Ok(Uuid::new_v4()),
+            |uid| {
+                from_value(uid.clone())
+                    .map_err(|_| AuthControllerError::InvalidApiKeyUid(uid.clone()))
+            },
+        )?;
+
+        let actions = value
+            .get("actions")
+            .map(|act| {
+                from_value(act.clone())
+                    .map_err(|_| AuthControllerError::InvalidApiKeyActions(act.clone()))
+            })
+            .ok_or(AuthControllerError::MissingParameter("actions"))??;
+
+        let indexes = value
+            .get("indexes")
+            .map(|ind| {
+                from_value(ind.clone())
+                    .map_err(|_| AuthControllerError::InvalidApiKeyIndexes(ind.clone()))
+            })
+            .ok_or(AuthControllerError::MissingParameter("indexes"))??;
+
+        let expires_at = value
+            .get("expiresAt")
+            .map(parse_expiration_date)
+            .ok_or(AuthControllerError::MissingParameter("expiresAt"))??;
+
+        let created_at = OffsetDateTime::now_utc();
+        let updated_at = created_at;
+
+        Ok(Self {
+            name,
+            description,
+            uid,
+            actions,
+            indexes,
+            expires_at,
+            created_at,
+            updated_at,
+        })
+    }
+
+    pub fn update_from_value(&mut self, value: Value) -> Result<()> {
+        if let Some(des) = value.get("description") {
+            let des = from_value(des.clone())
+                .map_err(|_| AuthControllerError::InvalidApiKeyDescription(des.clone()));
+            self.description = des?;
+        }
+
+        if let Some(des) = value.get("name") {
+            let des = from_value(des.clone())
+                .map_err(|_| AuthControllerError::InvalidApiKeyName(des.clone()));
+            self.name = des?;
+        }
+
+        if value.get("uid").is_some() {
+            return Err(AuthControllerError::ImmutableField("uid".to_string()));
+        }
+
+        if value.get("actions").is_some() {
+            return Err(AuthControllerError::ImmutableField("actions".to_string()));
+        }
+
+        if value.get("indexes").is_some() {
+            return Err(AuthControllerError::ImmutableField("indexes".to_string()));
+        }
+
+        if value.get("expiresAt").is_some() {
+            return Err(AuthControllerError::ImmutableField("expiresAt".to_string()));
+        }
+
+        if value.get("createdAt").is_some() {
+            return Err(AuthControllerError::ImmutableField("createdAt".to_string()));
+        }
+
+        if value.get("updatedAt").is_some() {
+            return Err(AuthControllerError::ImmutableField("updatedAt".to_string()));
+        }
+
+        self.updated_at = OffsetDateTime::now_utc();
+
+        Ok(())
+    }
+
+    pub(crate) fn default_admin() -> Self {
+        let now = OffsetDateTime::now_utc();
+        let uid = Uuid::new_v4();
+        Self {
+            name: Some("Default Admin API Key".to_string()),
+            description: Some("Use it for anything that is not a search operation. Caution! Do not expose it on a public frontend".to_string()),
+            uid,
+            actions: vec![Action::All],
+            indexes: vec![StarOr::Star],
+            expires_at: None,
+            created_at: now,
+            updated_at: now,
+        }
+    }
+
+    pub(crate) fn default_search() -> Self {
+        let now = OffsetDateTime::now_utc();
+        let uid = Uuid::new_v4();
+        Self {
+            name: Some("Default Search API Key".to_string()),
+            description: Some("Use it to search from the frontend".to_string()),
+            uid,
+            actions: vec![Action::Search],
+            indexes: vec![StarOr::Star],
+            expires_at: None,
+            created_at: now,
+            updated_at: now,
+        }
+    }
+}
+
+fn parse_expiration_date(value: &Value) -> Result<Option<OffsetDateTime>> {
+    match value {
+        Value::String(string) => OffsetDateTime::parse(string, &Rfc3339)
+            .or_else(|_| {
+                PrimitiveDateTime::parse(
+                    string,
+                    format_description!(
+                        "[year repr:full base:calendar]-[month repr:numerical]-[day]T[hour]:[minute]:[second]"
+                    ),
+                ).map(|datetime| datetime.assume_utc())
+            })
+            .or_else(|_| {
+                PrimitiveDateTime::parse(
+                    string,
+                    format_description!(
+                        "[year repr:full base:calendar]-[month repr:numerical]-[day] [hour]:[minute]:[second]"
+                    ),
+                ).map(|datetime| datetime.assume_utc())
+            })
+            .or_else(|_| {
+                    Date::parse(string, format_description!(
+                        "[year repr:full base:calendar]-[month repr:numerical]-[day]"
+                    )).map(|date| PrimitiveDateTime::new(date, time!(00:00)).assume_utc())
+            })
+            .map_err(|_| AuthControllerError::InvalidApiKeyExpiresAt(value.clone()))
+            // check if the key is already expired.
+            .and_then(|d| {
+                if d > OffsetDateTime::now_utc() {
+                    Ok(d)
+                } else {
+                    Err(AuthControllerError::InvalidApiKeyExpiresAt(value.clone()))
+                }
+            })
+            .map(Option::Some),
+        Value::Null => Ok(None),
+        _otherwise => Err(AuthControllerError::InvalidApiKeyExpiresAt(value.clone())),
+    }
+}

meilisearch-auth/src/lib.rs

@@ -0,0 +1,258 @@
+mod action;
+mod dump;
+pub mod error;
+mod key;
+mod store;
+
+use std::collections::{HashMap, HashSet};
+use std::ops::Deref;
+use std::path::Path;
+use std::sync::Arc;
+
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+use time::OffsetDateTime;
+use uuid::Uuid;
+
+pub use action::{actions, Action};
+use error::{AuthControllerError, Result};
+pub use key::Key;
+use meilisearch_types::star_or::StarOr;
+use store::generate_key_as_hexa;
+pub use store::open_auth_store_env;
+use store::HeedAuthStore;
+
+#[derive(Clone)]
+pub struct AuthController {
+    store: Arc<HeedAuthStore>,
+    master_key: Option<String>,
+}
+
+impl AuthController {
+    pub fn new(db_path: impl AsRef<Path>, master_key: &Option<String>) -> Result<Self> {
+        let store = HeedAuthStore::new(db_path)?;
+
+        if store.is_empty()? {
+            generate_default_keys(&store)?;
+        }
+
+        Ok(Self {
+            store: Arc::new(store),
+            master_key: master_key.clone(),
+        })
+    }
+
+    pub fn create_key(&self, value: Value) -> Result<Key> {
+        let key = Key::create_from_value(value)?;
+        match self.store.get_api_key(key.uid)? {
+            Some(_) => Err(AuthControllerError::ApiKeyAlreadyExists(
+                key.uid.to_string(),
+            )),
+            None => self.store.put_api_key(key),
+        }
+    }
+
+    pub fn update_key(&self, uid: Uuid, value: Value) -> Result<Key> {
+        let mut key = self.get_key(uid)?;
+        key.update_from_value(value)?;
+        self.store.put_api_key(key)
+    }
+
+    pub fn get_key(&self, uid: Uuid) -> Result<Key> {
+        self.store
+            .get_api_key(uid)?
+            .ok_or_else(|| AuthControllerError::ApiKeyNotFound(uid.to_string()))
+    }
+
+    pub fn get_optional_uid_from_encoded_key(&self, encoded_key: &[u8]) -> Result<Option<Uuid>> {
+        match &self.master_key {
+            Some(master_key) => self
+                .store
+                .get_uid_from_encoded_key(encoded_key, master_key.as_bytes()),
+            None => Ok(None),
+        }
+    }
+
+    pub fn get_uid_from_encoded_key(&self, encoded_key: &str) -> Result<Uuid> {
+        self.get_optional_uid_from_encoded_key(encoded_key.as_bytes())?
+            .ok_or_else(|| AuthControllerError::ApiKeyNotFound(encoded_key.to_string()))
+    }
+
+    pub fn get_key_filters(
+        &self,
+        uid: Uuid,
+        search_rules: Option<SearchRules>,
+    ) -> Result<AuthFilter> {
+        let mut filters = AuthFilter::default();
+        let key = self
+            .store
+            .get_api_key(uid)?
+            .ok_or_else(|| AuthControllerError::ApiKeyNotFound(uid.to_string()))?;
+
+        if !key.indexes.iter().any(|i| i == &StarOr::Star) {
+            filters.search_rules = match search_rules {
+                // Intersect search_rules with parent key authorized indexes.
+                Some(search_rules) => SearchRules::Map(
+                    key.indexes
+                        .into_iter()
+                        .filter_map(|index| {
+                            search_rules.get_index_search_rules(index.deref()).map(
+                                |index_search_rules| {
+                                    (String::from(index), Some(index_search_rules))
+                                },
+                            )
+                        })
+                        .collect(),
+                ),
+                None => SearchRules::Set(key.indexes.into_iter().map(String::from).collect()),
+            };
+        } else if let Some(search_rules) = search_rules {
+            filters.search_rules = search_rules;
+        }
+
+        filters.allow_index_creation = key
+            .actions
+            .iter()
+            .any(|&action| action == Action::IndexesAdd || action == Action::All);
+
+        Ok(filters)
+    }
+
+    pub fn list_keys(&self) -> Result<Vec<Key>> {
+        self.store.list_api_keys()
+    }
+
+    pub fn delete_key(&self, uid: Uuid) -> Result<()> {
+        if self.store.delete_api_key(uid)? {
+            Ok(())
+        } else {
+            Err(AuthControllerError::ApiKeyNotFound(uid.to_string()))
+        }
+    }
+
+    pub fn get_master_key(&self) -> Option<&String> {
+        self.master_key.as_ref()
+    }
+
+    /// Generate a valid key from a key id using the current master key.
+    /// Returns None if no master key has been set.
+    pub fn generate_key(&self, uid: Uuid) -> Option<String> {
+        self.master_key
+            .as_ref()
+            .map(|master_key| generate_key_as_hexa(uid, master_key.as_bytes()))
+    }
+
+    /// Check if the provided key is authorized to make a specific action
+    /// without checking if the key is valid.
+    pub fn is_key_authorized(
+        &self,
+        uid: Uuid,
+        action: Action,
+        index: Option<&str>,
+    ) -> Result<bool> {
+        match self
+            .store
+            // check if the key has access to all indexes.
+            .get_expiration_date(uid, action, None)?
+            .or(match index {
+                // else check if the key has access to the requested index.
+                Some(index) => {
+                    self.store
+                        .get_expiration_date(uid, action, Some(index.as_bytes()))?
+                }
+                // or to any index if no index has been requested.
+                None => self.store.prefix_first_expiration_date(uid, action)?,
+            }) {
+            // check expiration date.
+            Some(Some(exp)) => Ok(OffsetDateTime::now_utc() < exp),
+            // no expiration date.
+            Some(None) => Ok(true),
+            // action or index forbidden.
+            None => Ok(false),
+        }
+    }
+}
+
+pub struct AuthFilter {
+    pub search_rules: SearchRules,
+    pub allow_index_creation: bool,
+}
+
+impl Default for AuthFilter {
+    fn default() -> Self {
+        Self {
+            search_rules: SearchRules::default(),
+            allow_index_creation: true,
+        }
+    }
+}
+
+/// Transparent wrapper around a list of allowed indexes with the search rules to apply for each.
+#[derive(Debug, Serialize, Deserialize, Clone)]
+#[serde(untagged)]
+pub enum SearchRules {
+    Set(HashSet<String>),
+    Map(HashMap<String, Option<IndexSearchRules>>),
+}
+
+impl Default for SearchRules {
+    fn default() -> Self {
+        Self::Set(Some("*".to_string()).into_iter().collect())
+    }
+}
+
+impl SearchRules {
+    pub fn is_index_authorized(&self, index: &str) -> bool {
+        match self {
+            Self::Set(set) => set.contains("*") || set.contains(index),
+            Self::Map(map) => map.contains_key("*") || map.contains_key(index),
+        }
+    }
+
+    pub fn get_index_search_rules(&self, index: &str) -> Option<IndexSearchRules> {
+        match self {
+            Self::Set(set) => {
+                if set.contains("*") || set.contains(index) {
+                    Some(IndexSearchRules::default())
+                } else {
+                    None
+                }
+            }
+            Self::Map(map) => map
+                .get(index)
+                .or_else(|| map.get("*"))
+                .map(|isr| isr.clone().unwrap_or_default()),
+        }
+    }
+}
+
+impl IntoIterator for SearchRules {
+    type Item = (String, IndexSearchRules);
+    type IntoIter = Box<dyn Iterator<Item = Self::Item>>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        match self {
+            Self::Set(array) => {
+                Box::new(array.into_iter().map(|i| (i, IndexSearchRules::default())))
+            }
+            Self::Map(map) => {
+                Box::new(map.into_iter().map(|(i, isr)| (i, isr.unwrap_or_default())))
+            }
+        }
+    }
+}
+
+/// Contains the rules to apply on the top of the search query for a specific index.
+///
+/// filter: search filter to apply in addition to query filters.
+#[derive(Debug, Serialize, Deserialize, Default, Clone)]
+pub struct IndexSearchRules {
+    pub filter: Option<serde_json::Value>,
+}
+
+fn generate_default_keys(store: &HeedAuthStore) -> Result<()> {
+    store.put_api_key(Key::default_admin())?;
+    store.put_api_key(Key::default_search())?;
+
+    Ok(())
+}

meilisearch-auth/src/store.rs

@@ -0,0 +1,319 @@
+use std::borrow::Cow;
+use std::cmp::Reverse;
+use std::collections::HashSet;
+use std::convert::TryFrom;
+use std::convert::TryInto;
+use std::fs::create_dir_all;
+use std::ops::Deref;
+use std::path::Path;
+use std::str;
+use std::sync::Arc;
+
+use enum_iterator::IntoEnumIterator;
+use hmac::{Hmac, Mac};
+use meilisearch_types::star_or::StarOr;
+use milli::heed::types::{ByteSlice, DecodeIgnore, SerdeJson};
+use milli::heed::{Database, Env, EnvOpenOptions, RwTxn};
+use sha2::Sha256;
+use time::OffsetDateTime;
+use uuid::fmt::Hyphenated;
+use uuid::Uuid;
+
+use super::error::Result;
+use super::{Action, Key};
+
+const AUTH_STORE_SIZE: usize = 1_073_741_824; //1GiB
+const AUTH_DB_PATH: &str = "auth";
+const KEY_DB_NAME: &str = "api-keys";
+const KEY_ID_ACTION_INDEX_EXPIRATION_DB_NAME: &str = "keyid-action-index-expiration";
+
+pub type KeyId = Uuid;
+
+#[derive(Clone)]
+pub struct HeedAuthStore {
+    env: Arc<Env>,
+    keys: Database<ByteSlice, SerdeJson<Key>>,
+    action_keyid_index_expiration: Database<KeyIdActionCodec, SerdeJson<Option<OffsetDateTime>>>,
+    should_close_on_drop: bool,
+}
+
+impl Drop for HeedAuthStore {
+    fn drop(&mut self) {
+        if self.should_close_on_drop && Arc::strong_count(&self.env) == 1 {
+            self.env.as_ref().clone().prepare_for_closing();
+        }
+    }
+}
+
+pub fn open_auth_store_env(path: &Path) -> milli::heed::Result<milli::heed::Env> {
+    let mut options = EnvOpenOptions::new();
+    options.map_size(AUTH_STORE_SIZE); // 1GB
+    options.max_dbs(2);
+    options.open(path)
+}
+
+impl HeedAuthStore {
+    pub fn new(path: impl AsRef<Path>) -> Result<Self> {
+        let path = path.as_ref().join(AUTH_DB_PATH);
+        create_dir_all(&path)?;
+        let env = Arc::new(open_auth_store_env(path.as_ref())?);
+        let keys = env.create_database(Some(KEY_DB_NAME))?;
+        let action_keyid_index_expiration =
+            env.create_database(Some(KEY_ID_ACTION_INDEX_EXPIRATION_DB_NAME))?;
+        Ok(Self {
+            env,
+            keys,
+            action_keyid_index_expiration,
+            should_close_on_drop: true,
+        })
+    }
+
+    pub fn set_drop_on_close(&mut self, v: bool) {
+        self.should_close_on_drop = v;
+    }
+
+    pub fn is_empty(&self) -> Result<bool> {
+        let rtxn = self.env.read_txn()?;
+
+        Ok(self.keys.len(&rtxn)? == 0)
+    }
+
+    pub fn put_api_key(&self, key: Key) -> Result<Key> {
+        let uid = key.uid;
+        let mut wtxn = self.env.write_txn()?;
+
+        self.keys.put(&mut wtxn, uid.as_bytes(), &key)?;
+
+        // delete key from inverted database before refilling it.
+        self.delete_key_from_inverted_db(&mut wtxn, &uid)?;
+        // create inverted database.
+        let db = self.action_keyid_index_expiration;
+
+        let mut actions = HashSet::new();
+        for action in &key.actions {
+            match action {
+                Action::All => actions.extend(Action::into_enum_iter()),
+                Action::DocumentsAll => {
+                    actions.extend(
+                        [
+                            Action::DocumentsGet,
+                            Action::DocumentsDelete,
+                            Action::DocumentsAdd,
+                        ]
+                        .iter(),
+                    );
+                }
+                Action::IndexesAll => {
+                    actions.extend(
+                        [
+                            Action::IndexesAdd,
+                            Action::IndexesDelete,
+                            Action::IndexesGet,
+                            Action::IndexesUpdate,
+                        ]
+                        .iter(),
+                    );
+                }
+                Action::SettingsAll => {
+                    actions.extend([Action::SettingsGet, Action::SettingsUpdate].iter());
+                }
+                Action::DumpsAll => {
+                    actions.insert(Action::DumpsCreate);
+                }
+                Action::TasksAll => {
+                    actions.insert(Action::TasksGet);
+                }
+                Action::StatsAll => {
+                    actions.insert(Action::StatsGet);
+                }
+                other => {
+                    actions.insert(*other);
+                }
+            }
+        }
+
+        let no_index_restriction = key.indexes.contains(&StarOr::Star);
+        for action in actions {
+            if no_index_restriction {
+                // If there is no index restriction we put None.
+                db.put(&mut wtxn, &(&uid, &action, None), &key.expires_at)?;
+            } else {
+                // else we create a key for each index.
+                for index in key.indexes.iter() {
+                    db.put(
+                        &mut wtxn,
+                        &(&uid, &action, Some(index.deref().as_bytes())),
+                        &key.expires_at,
+                    )?;
+                }
+            }
+        }
+
+        wtxn.commit()?;
+
+        Ok(key)
+    }
+
+    pub fn get_api_key(&self, uid: Uuid) -> Result<Option<Key>> {
+        let rtxn = self.env.read_txn()?;
+        self.keys.get(&rtxn, uid.as_bytes()).map_err(|e| e.into())
+    }
+
+    pub fn get_uid_from_encoded_key(
+        &self,
+        encoded_key: &[u8],
+        master_key: &[u8],
+    ) -> Result<Option<Uuid>> {
+        let rtxn = self.env.read_txn()?;
+        let uid = self
+            .keys
+            .remap_data_type::<DecodeIgnore>()
+            .iter(&rtxn)?
+            .filter_map(|res| match res {
+                Ok((uid, _)) => {
+                    let (uid, _) = try_split_array_at(uid)?;
+                    let uid = Uuid::from_bytes(*uid);
+                    if generate_key_as_hexa(uid, master_key).as_bytes() == encoded_key {
+                        Some(uid)
+                    } else {
+                        None
+                    }
+                }
+                Err(_) => None,
+            })
+            .next();
+
+        Ok(uid)
+    }
+
+    pub fn delete_api_key(&self, uid: Uuid) -> Result<bool> {
+        let mut wtxn = self.env.write_txn()?;
+        let existing = self.keys.delete(&mut wtxn, uid.as_bytes())?;
+        self.delete_key_from_inverted_db(&mut wtxn, &uid)?;
+        wtxn.commit()?;
+
+        Ok(existing)
+    }
+
+    pub fn list_api_keys(&self) -> Result<Vec<Key>> {
+        let mut list = Vec::new();
+        let rtxn = self.env.read_txn()?;
+        for result in self.keys.remap_key_type::<DecodeIgnore>().iter(&rtxn)? {
+            let (_, content) = result?;
+            list.push(content);
+        }
+        list.sort_unstable_by_key(|k| Reverse(k.created_at));
+        Ok(list)
+    }
+
+    pub fn get_expiration_date(
+        &self,
+        uid: Uuid,
+        action: Action,
+        index: Option<&[u8]>,
+    ) -> Result<Option<Option<OffsetDateTime>>> {
+        let rtxn = self.env.read_txn()?;
+        let tuple = (&uid, &action, index);
+        Ok(self.action_keyid_index_expiration.get(&rtxn, &tuple)?)
+    }
+
+    pub fn prefix_first_expiration_date(
+        &self,
+        uid: Uuid,
+        action: Action,
+    ) -> Result<Option<Option<OffsetDateTime>>> {
+        let rtxn = self.env.read_txn()?;
+        let tuple = (&uid, &action, None);
+        let exp = self
+            .action_keyid_index_expiration
+            .prefix_iter(&rtxn, &tuple)?
+            .next()
+            .transpose()?
+            .map(|(_, expiration)| expiration);
+
+        Ok(exp)
+    }
+
+    fn delete_key_from_inverted_db(&self, wtxn: &mut RwTxn, key: &KeyId) -> Result<()> {
+        let mut iter = self
+            .action_keyid_index_expiration
+            .remap_types::<ByteSlice, DecodeIgnore>()
+            .prefix_iter_mut(wtxn, key.as_bytes())?;
+        while iter.next().transpose()?.is_some() {
+            // safety: we don't keep references from inside the LMDB database.
+            unsafe { iter.del_current()? };
+        }
+
+        Ok(())
+    }
+}
+
+/// Codec allowing to retrieve the expiration date of an action,
+/// optionally on a specific index, for a given key.
+pub struct KeyIdActionCodec;
+
+impl<'a> milli::heed::BytesDecode<'a> for KeyIdActionCodec {
+    type DItem = (KeyId, Action, Option<&'a [u8]>);
+
+    fn bytes_decode(bytes: &'a [u8]) -> Option<Self::DItem> {
+        let (key_id_bytes, action_bytes) = try_split_array_at(bytes)?;
+        let (action_bytes, index) = match try_split_array_at(action_bytes)? {
+            (action, []) => (action, None),
+            (action, index) => (action, Some(index)),
+        };
+        let key_id = Uuid::from_bytes(*key_id_bytes);
+        let action = Action::from_repr(u8::from_be_bytes(*action_bytes))?;
+
+        Some((key_id, action, index))
+    }
+}
+
+impl<'a> milli::heed::BytesEncode<'a> for KeyIdActionCodec {
+    type EItem = (&'a KeyId, &'a Action, Option<&'a [u8]>);
+
+    fn bytes_encode((key_id, action, index): &Self::EItem) -> Option<Cow<[u8]>> {
+        let mut bytes = Vec::new();
+
+        bytes.extend_from_slice(key_id.as_bytes());
+        let action_bytes = u8::to_be_bytes(action.repr());
+        bytes.extend_from_slice(&action_bytes);
+        if let Some(index) = index {
+            bytes.extend_from_slice(index);
+        }
+
+        Some(Cow::Owned(bytes))
+    }
+}
+
+pub fn generate_key_as_hexa(uid: Uuid, master_key: &[u8]) -> String {
+    // format uid as hyphenated allowing user to generate their own keys.
+    let mut uid_buffer = [0; Hyphenated::LENGTH];
+    let uid = uid.hyphenated().encode_lower(&mut uid_buffer);
+
+    // new_from_slice function never fail.
+    let mut mac = Hmac::<Sha256>::new_from_slice(master_key).unwrap();
+    mac.update(uid.as_bytes());
+
+    let result = mac.finalize();
+    format!("{:x}", result.into_bytes())
+}
+
+/// Divides one slice into two at an index, returns `None` if mid is out of bounds.
+pub fn try_split_at<T>(slice: &[T], mid: usize) -> Option<(&[T], &[T])> {
+    if mid <= slice.len() {
+        Some(slice.split_at(mid))
+    } else {
+        None
+    }
+}
+
+/// Divides one slice into an array and the tail at an index,
+/// returns `None` if `N` is out of bounds.
+pub fn try_split_array_at<T, const N: usize>(slice: &[T]) -> Option<(&[T; N], &[T])>
+where
+    [T; N]: for<'a> TryFrom<&'a [T]>,
+{
+    let (head, tail) = try_split_at(slice, N)?;
+    let head = head.try_into().ok()?;
+    Some((head, tail))
+}

meilisearch-core/Cargo.toml

@@ -1,57 +0,0 @@
-[package]
-name = "meilisearch-core"
-version = "0.14.0"
-license = "MIT"
-authors = ["Kerollmops <clement@meilisearch.com>"]
-edition = "2018"
-
-[dependencies]
-arc-swap = "0.4.5"
-bincode = "1.2.1"
-byteorder = "1.3.4"
-chrono = { version = "0.4.11", features = ["serde"] }
-compact_arena = "0.4.0"
-cow-utils = "0.1.2"
-crossbeam-channel = "0.4.2"
-deunicode = "1.1.0"
-either = "1.5.3"
-env_logger = "0.7.1"
-fst = "0.4.4"
-hashbrown = { version = "0.7.1", features = ["serde"] }
-heed = "0.8.0"
-indexmap = { version = "1.3.2", features = ["serde-1"] }
-intervaltree = "0.2.5"
-itertools = "0.9.0"
-levenshtein_automata = { version = "0.2.0", features = ["fst_automaton"] }
-log = "0.4.8"
-meilisearch-error = { path = "../meilisearch-error", version = "0.14.0" }
-meilisearch-schema = { path = "../meilisearch-schema", version = "0.14.0" }
-meilisearch-tokenizer = { path = "../meilisearch-tokenizer", version = "0.14.0" }
-meilisearch-types = { path = "../meilisearch-types", version = "0.14.0" }
-once_cell = "1.3.1"
-ordered-float = { version = "1.0.2", features = ["serde"] }
-pest = { git = "https://github.com/MarinPostma/pest.git", tag = "meilisearch-patch1" }
-pest_derive = "2.0"
-regex = "1.3.6"
-sdset = "0.4.0"
-serde = { version = "1.0.105", features = ["derive"] }
-serde_json = { version = "1.0.50", features = ["preserve_order"] }
-slice-group-by = "0.2.6"
-unicase = "2.6.0"
-zerocopy = "0.3.0"
-
-[dev-dependencies]
-assert_matches = "1.3.0"
-criterion = "0.3.1"
-csv = "1.1.3"
-rustyline = { version = "6.0.0", default-features = false }
-structopt = "0.3.12"
-tempfile = "3.1.0"
-termcolor = "1.1.0"
-
-[target.'cfg(unix)'.dev-dependencies]
-jemallocator = "0.3.2"
-
-[[bench]]
-name = "search_benchmark"
-harness = false

meilisearch-core/benches/search_benchmark.rs

@@ -1,108 +0,0 @@
-#[cfg(test)]
-#[macro_use]
-extern crate assert_matches;
-
-use std::error::Error;
-use std::fs::File;
-use std::io::BufReader;
-use std::iter;
-use std::path::Path;
-use std::sync::mpsc;
-
-use meilisearch_core::{Database, DatabaseOptions};
-use meilisearch_core::{ProcessedUpdateResult, UpdateStatus};
-use meilisearch_core::settings::{Settings, SettingsUpdate};
-use meilisearch_schema::Schema;
-use serde_json::Value;
-
-use criterion::{criterion_group, criterion_main, Criterion, BenchmarkId};
-
-fn prepare_database(path: &Path) -> Database {
-    let database = Database::open_or_create(path, DatabaseOptions::default()).unwrap();
-    let db = &database;
-
-    let (sender, receiver) = mpsc::sync_channel(100);
-    let update_fn = move |_name: &str, update: ProcessedUpdateResult| {
-        sender.send(update.update_id).unwrap()
-    };
-    let index = database.create_index("bench").unwrap();
-
-    database.set_update_callback(Box::new(update_fn));
-
-    db.main_write::<_, _, Box<dyn Error>>(|writer| {
-        index.main.put_schema(writer, &Schema::with_primary_key("id")).unwrap();
-        Ok(())
-    }).unwrap();
-
-    let settings_update: SettingsUpdate = {
-        let path = concat!(env!("CARGO_MANIFEST_DIR"), "/../datasets/movies/settings.json");
-        let file = File::open(path).unwrap();
-        let reader = BufReader::new(file);
-        let settings: Settings = serde_json::from_reader(reader).unwrap();
-        settings.to_update().unwrap()
-    };
-
-    db.update_write::<_, _, Box<dyn Error>>(|writer| {
-        let _update_id = index.settings_update(writer, settings_update).unwrap();
-        Ok(())
-    }).unwrap();
-
-    let mut additions = index.documents_addition();
-
-    let json: Value = {
-        let path = concat!(env!("CARGO_MANIFEST_DIR"), "/../datasets/movies/movies.json");
-        let movies_file = File::open(path).expect("find movies");
-        serde_json::from_reader(movies_file).unwrap()
-    };
-
-    let documents = json.as_array().unwrap();
-
-    for document in documents {
-        additions.update_document(document);
-    }
-
-    let update_id = db.update_write::<_, _, Box<dyn Error>>(|writer| {
-        let update_id = additions.finalize(writer).unwrap();
-        Ok(update_id)
-    }).unwrap();
-
-    // block until the transaction is processed
-    let _ = receiver.into_iter().find(|id| *id == update_id);
-
-    let update_reader = db.update_read_txn().unwrap();
-    let result = index.update_status(&update_reader, update_id).unwrap();
-    assert_matches!(result, Some(UpdateStatus::Processed { content }) if content.error.is_none());
-
-    database
-}
-
-pub fn criterion_benchmark(c: &mut Criterion) {
-    let dir = tempfile::tempdir().unwrap();
-    let database = prepare_database(dir.path());
-
-    let reader = database.main_read_txn().unwrap();
-    let index = database.open_index("bench").unwrap();
-
-    let mut count = 0;
-    let query = "I love paris ";
-
-    let iter = iter::from_fn(|| {
-        count += 1;
-        query.get(0..count)
-    });
-
-    let mut group = c.benchmark_group("searching in movies (19654 docs)");
-    group.sample_size(10);
-
-    for query in iter {
-        let bench_name = BenchmarkId::from_parameter(format!("{:?}", query));
-        group.bench_with_input(bench_name, &query, |b, query| b.iter(|| {
-            let builder = index.query_builder();
-            builder.query(&reader, query, 0..20).unwrap();
-        }));
-    }
-    group.finish();
-}
-
-criterion_group!(benches, criterion_benchmark);
-criterion_main!(benches);

meilisearch-core/examples/from_file.rs

@@ -1,470 +0,0 @@
-use std::collections::HashSet;
-use std::collections::btree_map::{BTreeMap, Entry};
-use std::error::Error;
-use std::io::{Read, Write};
-use std::iter::FromIterator;
-use std::path::{Path, PathBuf};
-use std::time::{Duration, Instant};
-use std::{fs, io, sync::mpsc};
-
-use rustyline::{Config, Editor};
-use serde::{Deserialize, Serialize};
-use structopt::StructOpt;
-use termcolor::{Color, ColorChoice, ColorSpec, StandardStream, WriteColor};
-
-use meilisearch_core::{Database, DatabaseOptions, Highlight, ProcessedUpdateResult};
-use meilisearch_core::settings::Settings;
-use meilisearch_schema::FieldId;
-
-#[cfg(target_os = "linux")]
-#[global_allocator]
-static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;
-
-#[derive(Debug, StructOpt)]
-struct IndexCommand {
-    /// The destination where the database must be created.
-    #[structopt(parse(from_os_str))]
-    database_path: PathBuf,
-
-    #[structopt(long, default_value = "default")]
-    index_uid: String,
-
-    /// The csv file path to index, you can also use `-` to specify the standard input.
-    #[structopt(parse(from_os_str))]
-    csv_data_path: PathBuf,
-
-    /// The path to the settings.
-    #[structopt(long, parse(from_os_str))]
-    settings: PathBuf,
-
-    #[structopt(long)]
-    update_group_size: Option<usize>,
-
-    #[structopt(long, parse(from_os_str))]
-    compact_to_path: Option<PathBuf>,
-}
-
-#[derive(Debug, StructOpt)]
-struct SearchCommand {
-    /// The path of the database to work with.
-    #[structopt(parse(from_os_str))]
-    database_path: PathBuf,
-
-    #[structopt(long, default_value = "default")]
-    index_uid: String,
-
-    /// Timeout after which the search will return results.
-    #[structopt(long)]
-    fetch_timeout_ms: Option<u64>,
-
-    /// The number of returned results
-    #[structopt(short, long, default_value = "10")]
-    number_results: usize,
-
-    /// The number of characters before and after the first match
-    #[structopt(short = "C", long, default_value = "35")]
-    char_context: usize,
-
-    /// A filter string that can be `!adult` or `adult` to
-    /// filter documents on this specfied field
-    #[structopt(short, long)]
-    filter: Option<String>,
-
-    /// Fields that must be displayed.
-    displayed_fields: Vec<String>,
-}
-
-#[derive(Debug, StructOpt)]
-struct ShowUpdatesCommand {
-    /// The path of the database to work with.
-    #[structopt(parse(from_os_str))]
-    database_path: PathBuf,
-
-    #[structopt(long, default_value = "default")]
-    index_uid: String,
-}
-
-#[derive(Debug, StructOpt)]
-enum Command {
-    Index(IndexCommand),
-    Search(SearchCommand),
-    ShowUpdates(ShowUpdatesCommand),
-}
-
-impl Command {
-    fn path(&self) -> &Path {
-        match self {
-            Command::Index(command) => &command.database_path,
-            Command::Search(command) => &command.database_path,
-            Command::ShowUpdates(command) => &command.database_path,
-        }
-    }
-}
-
-#[derive(Serialize, Deserialize)]
-#[serde(transparent)]
-struct Document(indexmap::IndexMap<String, String>);
-
-fn index_command(command: IndexCommand, database: Database) -> Result<(), Box<dyn Error>> {
-    let start = Instant::now();
-
-    let (sender, receiver) = mpsc::sync_channel(100);
-    let update_fn =
-        move |_name: &str, update: ProcessedUpdateResult| sender.send(update.update_id).unwrap();
-    let index = match database.open_index(&command.index_uid) {
-        Some(index) => index,
-        None => database.create_index(&command.index_uid).unwrap(),
-    };
-
-    database.set_update_callback(Box::new(update_fn));
-
-    let db = &database;
-
-    let settings = {
-        let string = fs::read_to_string(&command.settings)?;
-        let settings: Settings = serde_json::from_str(&string).unwrap();
-        settings.to_update().unwrap()
-    };
-
-    db.update_write(|w| index.settings_update(w, settings))?;
-
-    let mut rdr = if command.csv_data_path.as_os_str() == "-" {
-        csv::Reader::from_reader(Box::new(io::stdin()) as Box<dyn Read>)
-    } else {
-        let file = std::fs::File::open(command.csv_data_path)?;
-        csv::Reader::from_reader(Box::new(file) as Box<dyn Read>)
-    };
-
-    let mut raw_record = csv::StringRecord::new();
-    let headers = rdr.headers()?.clone();
-
-    let mut max_update_id = 0;
-    let mut i = 0;
-    let mut end_of_file = false;
-
-    while !end_of_file {
-        let mut additions = index.documents_addition();
-
-        loop {
-            end_of_file = !rdr.read_record(&mut raw_record)?;
-            if end_of_file {
-                break;
-            }
-
-            let document: Document = match raw_record.deserialize(Some(&headers)) {
-                Ok(document) => document,
-                Err(e) => {
-                    eprintln!("{:?}", e);
-                    continue;
-                }
-            };
-
-            additions.update_document(document);
-
-            print!("\rindexing document {}", i);
-            i += 1;
-
-            if let Some(group_size) = command.update_group_size {
-                if i % group_size == 0 {
-                    break;
-                }
-            }
-        }
-
-        println!();
-
-        let update_id = db.update_write(|w| additions.finalize(w))?;
-
-        println!("committing update...");
-        max_update_id = max_update_id.max(update_id);
-        println!("committed update {}", update_id);
-    }
-
-    println!("Waiting for update {}", max_update_id);
-    for id in receiver {
-        if id == max_update_id {
-            break;
-        }
-    }
-
-    println!(
-        "database created in {:.2?} at: {:?}",
-        start.elapsed(),
-        command.database_path
-    );
-
-    if let Some(path) = command.compact_to_path {
-        fs::create_dir_all(&path)?;
-        let start = Instant::now();
-        let _file = database.copy_and_compact_to_path(path.join("data.mdb"))?;
-        println!(
-            "database compacted in {:.2?} at: {:?}",
-            start.elapsed(),
-            path
-        );
-    }
-
-    Ok(())
-}
-
-fn display_highlights(text: &str, ranges: &[usize]) -> io::Result<()> {
-    let mut stdout = StandardStream::stdout(ColorChoice::Always);
-    let mut highlighted = false;
-
-    for range in ranges.windows(2) {
-        let [start, end] = match range {
-            [start, end] => [*start, *end],
-            _ => unreachable!(),
-        };
-        if highlighted {
-            stdout.set_color(
-                ColorSpec::new()
-                    .set_fg(Some(Color::Yellow))
-                    .set_underline(true),
-            )?;
-        }
-        write!(&mut stdout, "{}", &text[start..end])?;
-        stdout.reset()?;
-        highlighted = !highlighted;
-    }
-
-    Ok(())
-}
-
-fn char_to_byte_range(index: usize, length: usize, text: &str) -> (usize, usize) {
-    let mut byte_index = 0;
-    let mut byte_length = 0;
-
-    for (n, (i, c)) in text.char_indices().enumerate() {
-        if n == index {
-            byte_index = i;
-        }
-
-        if n + 1 == index + length {
-            byte_length = i - byte_index + c.len_utf8();
-            break;
-        }
-    }
-
-    (byte_index, byte_length)
-}
-
-fn create_highlight_areas(text: &str, highlights: &[Highlight]) -> Vec<usize> {
-    let mut byte_indexes = BTreeMap::new();
-
-    for highlight in highlights {
-        let char_index = highlight.char_index as usize;
-        let char_length = highlight.char_length as usize;
-        let (byte_index, byte_length) = char_to_byte_range(char_index, char_length, text);
-
-        match byte_indexes.entry(byte_index) {
-            Entry::Vacant(entry) => {
-                entry.insert(byte_length);
-            }
-            Entry::Occupied(mut entry) => {
-                if *entry.get() < byte_length {
-                    entry.insert(byte_length);
-                }
-            }
-        }
-    }
-
-    let mut title_areas = Vec::new();
-    title_areas.push(0);
-    for (byte_index, length) in byte_indexes {
-        title_areas.push(byte_index);
-        title_areas.push(byte_index + length);
-    }
-    title_areas.push(text.len());
-    title_areas.sort_unstable();
-    title_areas
-}
-
-/// note: matches must have been sorted by `char_index` and `char_length` before being passed.
-///
-/// ```no_run
-/// matches.sort_unstable_by_key(|m| (m.char_index, m.char_length));
-///
-/// let matches = matches.matches.iter().filter(|m| SchemaAttr::new(m.attribute) == attr).cloned();
-///
-/// let (text, matches) = crop_text(&text, matches, 35);
-/// ```
-fn crop_text(
-    text: &str,
-    highlights: impl IntoIterator<Item = Highlight>,
-    context: usize,
-) -> (String, Vec<Highlight>) {
-    let mut highlights = highlights.into_iter().peekable();
-
-    let char_index = highlights
-        .peek()
-        .map(|m| m.char_index as usize)
-        .unwrap_or(0);
-    let start = char_index.saturating_sub(context);
-    let text = text.chars().skip(start).take(context * 2).collect();
-
-    let highlights = highlights
-        .take_while(|m| (m.char_index as usize) + (m.char_length as usize) <= start + (context * 2))
-        .map(|highlight| Highlight {
-            char_index: highlight.char_index - start as u16,
-            ..highlight
-        })
-        .collect();
-
-    (text, highlights)
-}
-
-fn search_command(command: SearchCommand, database: Database) -> Result<(), Box<dyn Error>> {
-    let db = &database;
-    let index = database
-        .open_index(&command.index_uid)
-        .expect("Could not find index");
-
-    let reader = db.main_read_txn().unwrap();
-    let schema = index.main.schema(&reader)?;
-    reader.abort().unwrap();
-
-    let schema = schema.ok_or(meilisearch_core::Error::SchemaMissing)?;
-
-    let fields = command.displayed_fields.iter().map(String::as_str);
-    let fields = HashSet::from_iter(fields);
-
-    let config = Config::builder().auto_add_history(true).build();
-    let mut readline = Editor::<()>::with_config(config);
-    let _ = readline.load_history("query-history.txt");
-
-    for result in readline.iter("Searching for: ") {
-        match result {
-            Ok(query) => {
-                let start_total = Instant::now();
-
-                let reader = db.main_read_txn().unwrap();
-                let ref_index = &index;
-                let ref_reader = &reader;
-
-                let mut builder = index.query_builder();
-                if let Some(timeout) = command.fetch_timeout_ms {
-                    builder.with_fetch_timeout(Duration::from_millis(timeout));
-                }
-
-                if let Some(ref filter) = command.filter {
-                    let filter = filter.as_str();
-                    let (positive, filter) = if filter.chars().next() == Some('!') {
-                        (false, &filter[1..])
-                    } else {
-                        (true, filter)
-                    };
-
-                    let attr = schema
-                        .id(filter)
-                        .expect("Could not find filtered attribute");
-
-                    builder.with_filter(move |document_id| {
-                        let string: String = ref_index
-                            .document_attribute(ref_reader, document_id, attr)
-                            .unwrap()
-                            .unwrap();
-                        (string == "true") == positive
-                    });
-                }
-
-                let result = builder.query(ref_reader, Some(&query), 0..command.number_results)?;
-
-                let mut retrieve_duration = Duration::default();
-
-                let number_of_documents = result.documents.len();
-                for mut doc in result.documents {
-                    doc.highlights
-                        .sort_unstable_by_key(|m| (m.char_index, m.char_length));
-
-                    let start_retrieve = Instant::now();
-                    let result = index.document::<Document>(&reader, Some(&fields), doc.id);
-                    retrieve_duration += start_retrieve.elapsed();
-
-                    match result {
-                        Ok(Some(document)) => {
-                            println!("raw-id: {:?}", doc.id);
-                            for (name, text) in document.0 {
-                                print!("{}: ", name);
-
-                                let attr = schema.id(&name).unwrap();
-                                let highlights = doc
-                                    .highlights
-                                    .iter()
-                                    .filter(|m| FieldId::new(m.attribute) == attr)
-                                    .cloned();
-                                let (text, highlights) =
-                                    crop_text(&text, highlights, command.char_context);
-                                let areas = create_highlight_areas(&text, &highlights);
-                                display_highlights(&text, &areas)?;
-                                println!();
-                            }
-                        }
-                        Ok(None) => eprintln!("missing document"),
-                        Err(e) => eprintln!("{}", e),
-                    }
-
-                    let mut matching_attributes = HashSet::new();
-                    for highlight in doc.highlights {
-                        let attr = FieldId::new(highlight.attribute);
-                        let name = schema.name(attr);
-                        matching_attributes.insert(name);
-                    }
-
-                    let matching_attributes = Vec::from_iter(matching_attributes);
-                    println!("matching in: {:?}", matching_attributes);
-
-                    println!();
-                }
-
-                eprintln!(
-                    "whole documents fields retrieve took {:.2?}",
-                    retrieve_duration
-                );
-                eprintln!(
-                    "===== Found {} results in {:.2?} =====",
-                    number_of_documents,
-                    start_total.elapsed()
-                );
-            }
-            Err(err) => {
-                println!("Error: {:?}", err);
-                break;
-            }
-        }
-    }
-
-    readline.save_history("query-history.txt").unwrap();
-
-    Ok(())
-}
-
-fn show_updates_command(
-    command: ShowUpdatesCommand,
-    database: Database,
-) -> Result<(), Box<dyn Error>> {
-    let db = &database;
-    let index = database
-        .open_index(&command.index_uid)
-        .expect("Could not find index");
-
-    let reader = db.update_read_txn().unwrap();
-    let updates = index.all_updates_status(&reader)?;
-    println!("{:#?}", updates);
-    reader.abort().unwrap();
-
-    Ok(())
-}
-
-fn main() -> Result<(), Box<dyn Error>> {
-    env_logger::init();
-
-    let opt = Command::from_args();
-    let database = Database::open_or_create(opt.path(), DatabaseOptions::default())?;
-
-    match opt {
-        Command::Index(command) => index_command(command, database),
-        Command::Search(command) => search_command(command, database),
-        Command::ShowUpdates(command) => show_updates_command(command, database),
-    }
-}

meilisearch-core/src/automaton/dfa.rs

@@ -1,53 +0,0 @@
-use levenshtein_automata::{LevenshteinAutomatonBuilder as LevBuilder, DFA};
-use once_cell::sync::OnceCell;
-
-static LEVDIST0: OnceCell<LevBuilder> = OnceCell::new();
-static LEVDIST1: OnceCell<LevBuilder> = OnceCell::new();
-static LEVDIST2: OnceCell<LevBuilder> = OnceCell::new();
-
-#[derive(Copy, Clone)]
-enum PrefixSetting {
-    Prefix,
-    NoPrefix,
-}
-
-fn build_dfa_with_setting(query: &str, setting: PrefixSetting) -> DFA {
-    use PrefixSetting::{NoPrefix, Prefix};
-
-    match query.len() {
-        0..=4 => {
-            let builder = LEVDIST0.get_or_init(|| LevBuilder::new(0, true));
-            match setting {
-                Prefix => builder.build_prefix_dfa(query),
-                NoPrefix => builder.build_dfa(query),
-            }
-        }
-        5..=8 => {
-            let builder = LEVDIST1.get_or_init(|| LevBuilder::new(1, true));
-            match setting {
-                Prefix => builder.build_prefix_dfa(query),
-                NoPrefix => builder.build_dfa(query),
-            }
-        }
-        _ => {
-            let builder = LEVDIST2.get_or_init(|| LevBuilder::new(2, true));
-            match setting {
-                Prefix => builder.build_prefix_dfa(query),
-                NoPrefix => builder.build_dfa(query),
-            }
-        }
-    }
-}
-
-pub fn build_prefix_dfa(query: &str) -> DFA {
-    build_dfa_with_setting(query, PrefixSetting::Prefix)
-}
-
-pub fn build_dfa(query: &str) -> DFA {
-    build_dfa_with_setting(query, PrefixSetting::NoPrefix)
-}
-
-pub fn build_exact_dfa(query: &str) -> DFA {
-    let builder = LEVDIST0.get_or_init(|| LevBuilder::new(0, true));
-    builder.build_dfa(query)
-}

meilisearch-core/src/automaton/mod.rs

@@ -1,15 +0,0 @@
-mod dfa;
-
-use meilisearch_tokenizer::is_cjk;
-
-pub use self::dfa::{build_dfa, build_prefix_dfa, build_exact_dfa};
-
-pub fn normalize_str(string: &str) -> String {
-    let mut string = string.to_lowercase();
-
-    if !string.contains(is_cjk) {
-        string = deunicode::deunicode_with_tofu(&string, "");
-    }
-
-    string
-}

meilisearch-core/src/bucket_sort.rs

@@ -1,661 +0,0 @@
-use std::borrow::Cow;
-use std::collections::HashMap;
-use std::mem;
-use std::ops::Deref;
-use std::ops::Range;
-use std::rc::Rc;
-use std::sync::atomic::{AtomicUsize, Ordering};
-use std::time::Instant;
-use std::fmt;
-
-use compact_arena::{SmallArena, Idx32, mk_arena};
-use log::{debug, error};
-use sdset::{Set, SetBuf, exponential_search, SetOperation, Counter, duo::OpBuilder};
-use slice_group_by::{GroupBy, GroupByMut};
-
-use meilisearch_types::DocIndex;
-
-use crate::criterion::{Criteria, Context, ContextMut};
-use crate::distinct_map::{BufferedDistinctMap, DistinctMap};
-use crate::raw_document::RawDocument;
-use crate::{database::MainT, reordered_attrs::ReorderedAttrs};
-use crate::{store, Document, DocumentId, MResult, Index, RankedMap, MainReader, Error};
-use crate::query_tree::{create_query_tree, traverse_query_tree};
-use crate::query_tree::{Operation, QueryResult, QueryKind, QueryId, PostingsKey};
-use crate::query_tree::Context as QTContext;
-
-#[derive(Debug, Default)]
-pub struct SortResult {
-    pub documents: Vec<Document>,
-    pub nb_hits: usize,
-    pub exhaustive_nb_hit: bool,
-    pub facets: Option<HashMap<String, HashMap<String, usize>>>,
-    pub exhaustive_facets_count: Option<bool>,
-}
-
-#[allow(clippy::too_many_arguments)]
-pub fn bucket_sort<'c, FI>(
-    reader: &heed::RoTxn<MainT>,
-    query: &str,
-    range: Range<usize>,
-    facets_docids: Option<SetBuf<DocumentId>>,
-    facet_count_docids: Option<HashMap<String, HashMap<String, (&str, Cow<Set<DocumentId>>)>>>,
-    filter: Option<FI>,
-    criteria: Criteria<'c>,
-    searchable_attrs: Option<ReorderedAttrs>,
-    index: &Index,
-) -> MResult<SortResult>
-where
-    FI: Fn(DocumentId) -> bool,
-{
-    // We delegate the filter work to the distinct query builder,
-    // specifying a distinct rule that has no effect.
-    if filter.is_some() {
-        let distinct = |_| None;
-        let distinct_size = 1;
-        return bucket_sort_with_distinct(
-            reader,
-            query,
-            range,
-            facets_docids,
-            facet_count_docids,
-            filter,
-            distinct,
-            distinct_size,
-            criteria,
-            searchable_attrs,
-            index,
-        );
-    }
-
-    let mut result = SortResult::default();
-
-    let words_set = index.main.words_fst(reader)?;
-    let stop_words = index.main.stop_words_fst(reader)?;
-
-    let context = QTContext {
-        words_set,
-        stop_words,
-        synonyms: index.synonyms,
-        postings_lists: index.postings_lists,
-        prefix_postings_lists: index.prefix_postings_lists_cache,
-    };
-
-    let (operation, mapping) = create_query_tree(reader, &context, query)?;
-    debug!("operation:\n{:?}", operation);
-    debug!("mapping:\n{:?}", mapping);
-
-    fn recurs_operation<'o>(map: &mut HashMap<QueryId, &'o QueryKind>, operation: &'o Operation) {
-        match operation {
-            Operation::And(ops) => ops.iter().for_each(|op| recurs_operation(map, op)),
-            Operation::Or(ops) => ops.iter().for_each(|op| recurs_operation(map, op)),
-            Operation::Query(query) => { map.insert(query.id, &query.kind); },
-        }
-    }
-
-    let mut queries_kinds = HashMap::new();
-    recurs_operation(&mut queries_kinds, &operation);
-
-    let QueryResult { mut docids, queries } = traverse_query_tree(reader, &context, &operation)?;
-    debug!("found {} documents", docids.len());
-    debug!("number of postings {:?}", queries.len());
-
-    if let Some(facets_docids) = facets_docids {
-        let intersection = sdset::duo::OpBuilder::new(docids.as_ref(), facets_docids.as_set())
-            .intersection()
-            .into_set_buf();
-        docids = Cow::Owned(intersection);
-    }
-
-    if let Some(f) = facet_count_docids {
-        // hardcoded value, until approximation optimization
-        result.exhaustive_facets_count = Some(true);
-        result.facets = Some(facet_count(f, &docids));
-    }
-
-    let before = Instant::now();
-    mk_arena!(arena);
-    let mut bare_matches = cleanup_bare_matches(&mut arena, &docids, queries);
-    debug!("matches cleaned in {:.02?}", before.elapsed());
-
-    let before_bucket_sort = Instant::now();
-
-    let before_raw_documents_building = Instant::now();
-    let mut raw_documents = Vec::new();
-    for bare_matches in bare_matches.linear_group_by_key_mut(|sm| sm.document_id) {
-        let raw_document = RawDocument::new(bare_matches, &mut arena, searchable_attrs.as_ref());
-        raw_documents.push(raw_document);
-    }
-    debug!("creating {} candidates documents took {:.02?}",
-        raw_documents.len(),
-        before_raw_documents_building.elapsed(),
-    );
-
-    let before_criterion_loop = Instant::now();
-    let proximity_count = AtomicUsize::new(0);
-
-    let mut groups = vec![raw_documents.as_mut_slice()];
-
-    'criteria: for criterion in criteria.as_ref() {
-        let tmp_groups = mem::replace(&mut groups, Vec::new());
-        let mut documents_seen = 0;
-
-        for mut group in tmp_groups {
-            let before_criterion_preparation = Instant::now();
-
-            let ctx = ContextMut {
-                reader,
-                postings_lists: &mut arena,
-                query_mapping: &mapping,
-                documents_fields_counts_store: index.documents_fields_counts,
-            };
-
-            criterion.prepare(ctx, &mut group)?;
-            debug!("{:?} preparation took {:.02?}", criterion.name(), before_criterion_preparation.elapsed());
-
-            let ctx = Context {
-                postings_lists: &arena,
-                query_mapping: &mapping,
-            };
-
-            let before_criterion_sort = Instant::now();
-            group.sort_unstable_by(|a, b| criterion.evaluate(&ctx, a, b));
-            debug!("{:?} evaluation took {:.02?}", criterion.name(), before_criterion_sort.elapsed());
-
-            for group in group.binary_group_by_mut(|a, b| criterion.eq(&ctx, a, b)) {
-                debug!("{:?} produced a group of size {}", criterion.name(), group.len());
-
-                documents_seen += group.len();
-                groups.push(group);
-
-                // we have sort enough documents if the last document sorted is after
-                // the end of the requested range, we can continue to the next criterion
-                if documents_seen >= range.end {
-                    continue 'criteria;
-                }
-            }
-        }
-    }
-
-    debug!("criterion loop took {:.02?}", before_criterion_loop.elapsed());
-    debug!("proximity evaluation called {} times", proximity_count.load(Ordering::Relaxed));
-
-    let schema = index.main.schema(reader)?.ok_or(Error::SchemaMissing)?;
-    let iter = raw_documents.into_iter().skip(range.start).take(range.len());
-    let iter = iter.map(|rd| Document::from_raw(rd, &queries_kinds, &arena, searchable_attrs.as_ref(), &schema));
-    let documents = iter.collect();
-
-    debug!("bucket sort took {:.02?}", before_bucket_sort.elapsed());
-
-    result.documents = documents;
-    result.nb_hits = docids.len();
-
-    Ok(result)
-}
-
-#[allow(clippy::too_many_arguments)]
-pub fn bucket_sort_with_distinct<'c, FI, FD>(
-    reader: &heed::RoTxn<MainT>,
-    query: &str,
-    range: Range<usize>,
-    facets_docids: Option<SetBuf<DocumentId>>,
-    facet_count_docids: Option<HashMap<String, HashMap<String, (&str, Cow<Set<DocumentId>>)>>>,
-    filter: Option<FI>,
-    distinct: FD,
-    distinct_size: usize,
-    criteria: Criteria<'c>,
-    searchable_attrs: Option<ReorderedAttrs>,
-    index: &Index,
-) -> MResult<SortResult>
-where
-    FI: Fn(DocumentId) -> bool,
-    FD: Fn(DocumentId) -> Option<u64>,
-{
-    let mut result = SortResult::default();
-
-    let words_set = index.main.words_fst(reader)?;
-    let stop_words = index.main.stop_words_fst(reader)?;
-
-    let context = QTContext {
-        words_set,
-        stop_words,
-        synonyms: index.synonyms,
-        postings_lists: index.postings_lists,
-        prefix_postings_lists: index.prefix_postings_lists_cache,
-    };
-
-    let (operation, mapping) = create_query_tree(reader, &context, query)?;
-    debug!("operation:\n{:?}", operation);
-    debug!("mapping:\n{:?}", mapping);
-
-    fn recurs_operation<'o>(map: &mut HashMap<QueryId, &'o QueryKind>, operation: &'o Operation) {
-        match operation {
-            Operation::And(ops) => ops.iter().for_each(|op| recurs_operation(map, op)),
-            Operation::Or(ops) => ops.iter().for_each(|op| recurs_operation(map, op)),
-            Operation::Query(query) => { map.insert(query.id, &query.kind); },
-        }
-    }
-
-    let mut queries_kinds = HashMap::new();
-    recurs_operation(&mut queries_kinds, &operation);
-
-    let QueryResult { mut docids, queries } = traverse_query_tree(reader, &context, &operation)?;
-    debug!("found {} documents", docids.len());
-    debug!("number of postings {:?}", queries.len());
-
-    if let Some(facets_docids) = facets_docids {
-        let intersection = OpBuilder::new(docids.as_ref(), facets_docids.as_set())
-            .intersection()
-            .into_set_buf();
-        docids = Cow::Owned(intersection);
-    }
-
-    if let Some(f) = facet_count_docids {
-        // hardcoded value, until approximation optimization
-        result.exhaustive_facets_count = Some(true);
-        result.facets = Some(facet_count(f, &docids));
-    }
-
-    let before = Instant::now();
-    mk_arena!(arena);
-    let mut bare_matches = cleanup_bare_matches(&mut arena, &docids, queries);
-    debug!("matches cleaned in {:.02?}", before.elapsed());
-
-    let before_raw_documents_building = Instant::now();
-    let mut raw_documents = Vec::new();
-    for bare_matches in bare_matches.linear_group_by_key_mut(|sm| sm.document_id) {
-        let raw_document = RawDocument::new(bare_matches, &mut arena, searchable_attrs.as_ref());
-        raw_documents.push(raw_document);
-    }
-    debug!("creating {} candidates documents took {:.02?}",
-        raw_documents.len(),
-        before_raw_documents_building.elapsed(),
-    );
-
-    let mut groups = vec![raw_documents.as_mut_slice()];
-    let mut key_cache = HashMap::new();
-
-    let mut filter_map = HashMap::new();
-    // these two variables informs on the current distinct map and
-    // on the raw offset of the start of the group where the
-    // range.start bound is located according to the distinct function
-    let mut distinct_map = DistinctMap::new(distinct_size);
-    let mut distinct_raw_offset = 0;
-
-    'criteria: for criterion in criteria.as_ref() {
-        let tmp_groups = mem::replace(&mut groups, Vec::new());
-        let mut buf_distinct = BufferedDistinctMap::new(&mut distinct_map);
-        let mut documents_seen = 0;
-
-        for mut group in tmp_groups {
-            // if this group does not overlap with the requested range,
-            // push it without sorting and splitting it
-            if documents_seen + group.len() < distinct_raw_offset {
-                documents_seen += group.len();
-                groups.push(group);
-                continue;
-            }
-
-            let ctx = ContextMut {
-                reader,
-                postings_lists: &mut arena,
-                query_mapping: &mapping,
-                documents_fields_counts_store: index.documents_fields_counts,
-            };
-
-            let before_criterion_preparation = Instant::now();
-            criterion.prepare(ctx, &mut group)?;
-            debug!("{:?} preparation took {:.02?}", criterion.name(), before_criterion_preparation.elapsed());
-
-            let ctx = Context {
-                postings_lists: &arena,
-                query_mapping: &mapping,
-            };
-
-            let before_criterion_sort = Instant::now();
-            group.sort_unstable_by(|a, b| criterion.evaluate(&ctx, a, b));
-            debug!("{:?} evaluation took {:.02?}", criterion.name(), before_criterion_sort.elapsed());
-
-            for group in group.binary_group_by_mut(|a, b| criterion.eq(&ctx, a, b)) {
-                // we must compute the real distinguished len of this sub-group
-                for document in group.iter() {
-                    let filter_accepted = match &filter {
-                        Some(filter) => {
-                            let entry = filter_map.entry(document.id);
-                            *entry.or_insert_with(|| (filter)(document.id))
-                        }
-                        None => true,
-                    };
-
-                    if filter_accepted {
-                        let entry = key_cache.entry(document.id);
-                        let key = entry.or_insert_with(|| (distinct)(document.id).map(Rc::new));
-
-                        match key.clone() {
-                            Some(key) => buf_distinct.register(key),
-                            None => buf_distinct.register_without_key(),
-                        };
-                    }
-
-                    // the requested range end is reached: stop computing distinct
-                    if buf_distinct.len() >= range.end {
-                        break;
-                    }
-                }
-
-                documents_seen += group.len();
-                groups.push(group);
-
-                // if this sub-group does not overlap with the requested range
-                // we must update the distinct map and its start index
-                if buf_distinct.len() < range.start {
-                    buf_distinct.transfert_to_internal();
-                    distinct_raw_offset = documents_seen;
-                }
-
-                // we have sort enough documents if the last document sorted is after
-                // the end of the requested range, we can continue to the next criterion
-                if buf_distinct.len() >= range.end {
-                    continue 'criteria;
-                }
-            }
-        }
-    }
-
-    // once we classified the documents related to the current
-    // automatons we save that as the next valid result
-    let mut seen = BufferedDistinctMap::new(&mut distinct_map);
-    let schema = index.main.schema(reader)?.ok_or(Error::SchemaMissing)?;
-
-    let mut documents = Vec::with_capacity(range.len());
-    for raw_document in raw_documents.into_iter().skip(distinct_raw_offset) {
-        let filter_accepted = match &filter {
-            Some(_) => filter_map.remove(&raw_document.id).unwrap_or_else(|| {
-                error!("error during filtering: expected value for document id {}", &raw_document.id.0);
-                Default::default()
-            }),
-            None => true,
-        };
-
-        if filter_accepted {
-            let key = key_cache.remove(&raw_document.id).unwrap_or_else(|| {
-                error!("error during distinct: expected value for document id {}", &raw_document.id.0);
-                Default::default()
-            });
-            let distinct_accepted = match key {
-                Some(key) => seen.register(key),
-                None => seen.register_without_key(),
-            };
-
-            if distinct_accepted && seen.len() > range.start {
-                documents.push(Document::from_raw(raw_document, &queries_kinds, &arena, searchable_attrs.as_ref(), &schema));
-                if documents.len() == range.len() {
-                    break;
-                }
-            }
-        }
-    }
-    result.documents = documents;
-    result.nb_hits = docids.len();
-
-    Ok(result)
-}
-
-fn cleanup_bare_matches<'tag, 'txn>(
-    arena: &mut SmallArena<'tag, PostingsListView<'txn>>,
-    docids: &Set<DocumentId>,
-    queries: HashMap<PostingsKey, Cow<'txn, Set<DocIndex>>>,
-) -> Vec<BareMatch<'tag>>
-{
-    let docidslen = docids.len() as f32;
-    let mut bare_matches = Vec::new();
-
-    for (PostingsKey { query, input, distance, is_exact }, matches) in queries {
-        let postings_list_view = PostingsListView::original(Rc::from(input), Rc::new(matches));
-        let pllen = postings_list_view.len() as f32;
-
-        if docidslen / pllen >= 0.8 {
-            let mut offset = 0;
-            for matches in postings_list_view.linear_group_by_key(|m| m.document_id) {
-                let document_id = matches[0].document_id;
-                if docids.contains(&document_id) {
-                    let range = postings_list_view.range(offset, matches.len());
-                    let posting_list_index = arena.add(range);
-
-                    let bare_match = BareMatch {
-                        document_id,
-                        query_index: query.id,
-                        distance,
-                        is_exact,
-                        postings_list: posting_list_index,
-                    };
-
-                    bare_matches.push(bare_match);
-                }
-
-                offset += matches.len();
-            }
-
-        } else {
-            let mut offset = 0;
-            for id in docids.as_slice() {
-                let di = DocIndex { document_id: *id, ..DocIndex::default() };
-                let pos = exponential_search(&postings_list_view[offset..], &di).unwrap_or_else(|x| x);
-
-                offset += pos;
-
-                let group = postings_list_view[offset..]
-                    .linear_group_by_key(|m| m.document_id)
-                    .next()
-                    .filter(|matches| matches[0].document_id == *id);
-
-                if let Some(matches) = group {
-                    let range = postings_list_view.range(offset, matches.len());
-                    let posting_list_index = arena.add(range);
-
-                    let bare_match = BareMatch {
-                        document_id: *id,
-                        query_index: query.id,
-                        distance,
-                        is_exact,
-                        postings_list: posting_list_index,
-                    };
-
-                    bare_matches.push(bare_match);
-                }
-            }
-        }
-    }
-
-    let before_raw_documents_presort = Instant::now();
-    bare_matches.sort_unstable_by_key(|sm| sm.document_id);
-    debug!("sort by documents ids took {:.02?}", before_raw_documents_presort.elapsed());
-
-    bare_matches
-}
-
-pub struct BareMatch<'tag> {
-    pub document_id: DocumentId,
-    pub query_index: usize,
-    pub distance: u8,
-    pub is_exact: bool,
-    pub postings_list: Idx32<'tag>,
-}
-
-impl fmt::Debug for BareMatch<'_> {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        f.debug_struct("BareMatch")
-            .field("document_id", &self.document_id)
-            .field("query_index", &self.query_index)
-            .field("distance", &self.distance)
-            .field("is_exact", &self.is_exact)
-            .finish()
-    }
-}
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
-pub struct SimpleMatch {
-    pub query_index: usize,
-    pub distance: u8,
-    pub attribute: u16,
-    pub word_index: u16,
-    pub is_exact: bool,
-}
-
-#[derive(Clone)]
-pub enum PostingsListView<'txn> {
-    Original {
-        input: Rc<[u8]>,
-        postings_list: Rc<Cow<'txn, Set<DocIndex>>>,
-        offset: usize,
-        len: usize,
-    },
-    Rewritten {
-        input: Rc<[u8]>,
-        postings_list: SetBuf<DocIndex>,
-    },
-}
-
-impl fmt::Debug for PostingsListView<'_> {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        f.debug_struct("PostingsListView")
-            .field("input", &std::str::from_utf8(&self.input()).unwrap())
-            .field("postings_list", &self.as_ref())
-            .finish()
-    }
-}
-
-impl<'txn> PostingsListView<'txn> {
-    pub fn original(input: Rc<[u8]>, postings_list: Rc<Cow<'txn, Set<DocIndex>>>) -> PostingsListView<'txn> {
-        let len = postings_list.len();
-        PostingsListView::Original { input, postings_list, offset: 0, len }
-    }
-
-    pub fn rewritten(input: Rc<[u8]>, postings_list: SetBuf<DocIndex>) -> PostingsListView<'txn> {
-        PostingsListView::Rewritten { input, postings_list }
-    }
-
-    pub fn rewrite_with(&mut self, postings_list: SetBuf<DocIndex>) {
-        let input = match self {
-            PostingsListView::Original { input, .. } => input.clone(),
-            PostingsListView::Rewritten { input, .. } => input.clone(),
-        };
-        *self = PostingsListView::rewritten(input, postings_list);
-    }
-
-    pub fn len(&self) -> usize {
-        match self {
-            PostingsListView::Original { len, .. } => *len,
-            PostingsListView::Rewritten { postings_list, .. } => postings_list.len(),
-        }
-    }
-
-    pub fn input(&self) -> &[u8] {
-        match self {
-            PostingsListView::Original { ref input, .. } => input,
-            PostingsListView::Rewritten { ref input, .. } => input,
-        }
-    }
-
-    pub fn range(&self, range_offset: usize, range_len: usize) -> PostingsListView<'txn> {
-        match self {
-            PostingsListView::Original { input, postings_list, offset, len } => {
-                assert!(range_offset + range_len <= *len);
-                PostingsListView::Original {
-                    input: input.clone(),
-                    postings_list: postings_list.clone(),
-                    offset: offset + range_offset,
-                    len: range_len,
-                }
-            },
-            PostingsListView::Rewritten { .. } => {
-                panic!("Cannot create a range on a rewritten postings list view");
-            }
-        }
-    }
-}
-
-impl AsRef<Set<DocIndex>> for PostingsListView<'_> {
-    fn as_ref(&self) -> &Set<DocIndex> {
-        self
-    }
-}
-
-impl Deref for PostingsListView<'_> {
-    type Target = Set<DocIndex>;
-
-    fn deref(&self) -> &Set<DocIndex> {
-        match *self {
-            PostingsListView::Original { ref postings_list, offset, len, .. } => {
-                Set::new_unchecked(&postings_list[offset..offset + len])
-            },
-            PostingsListView::Rewritten { ref postings_list, .. } => postings_list,
-        }
-    }
-}
-
-/// sorts documents ids according to user defined ranking rules.
-pub fn placeholder_document_sort(
-    document_ids: &mut [DocumentId],
-    index: &store::Index,
-    reader: &MainReader,
-    ranked_map: &RankedMap
-) -> MResult<()> {
-    use crate::settings::RankingRule;
-    use std::cmp::Ordering;
-
-    enum SortOrder {
-        Asc,
-        Desc,
-    }
-
-    if let Some(ranking_rules) = index.main.ranking_rules(reader)? {
-        let schema = index.main.schema(reader)?
-            .ok_or(Error::SchemaMissing)?;
-
-        // Select custom rules from ranking rules, and map them to custom rules
-        // containing a field_id
-        let ranking_rules = ranking_rules.iter().filter_map(|r|
-            match r {
-                RankingRule::Asc(name) => schema.id(name).map(|f| (f, SortOrder::Asc)),
-                RankingRule::Desc(name) => schema.id(name).map(|f| (f, SortOrder::Desc)),
-                _ => None,
-            }).collect::<Vec<_>>();
-
-        document_ids.sort_unstable_by(|a, b| {
-            for (field_id, order) in &ranking_rules {
-                let a_value = ranked_map.get(*a, *field_id);
-                let b_value = ranked_map.get(*b, *field_id);
-                let (a, b) = match order {
-                    SortOrder::Asc => (a_value, b_value),
-                    SortOrder::Desc => (b_value, a_value),
-                };
-                match a.cmp(&b) {
-                    Ordering::Equal => continue,
-                    ordering => return ordering,
-                }
-            }
-            Ordering::Equal
-        });
-    }
-    Ok(())
-}
-
-/// For each entry in facet_docids, calculates the number of documents in the intersection with candidate_docids.
-pub fn facet_count(
-    facet_docids: HashMap<String, HashMap<String, (&str, Cow<Set<DocumentId>>)>>,
-    candidate_docids: &Set<DocumentId>,
-) -> HashMap<String, HashMap<String, usize>> {
-    let mut facets_counts = HashMap::with_capacity(facet_docids.len());
-    for (key, doc_map) in facet_docids {
-        let mut count_map = HashMap::with_capacity(doc_map.len());
-        for (_, (value, docids)) in doc_map {
-            let mut counter = Counter::new();
-            let op = OpBuilder::new(docids.as_ref(), candidate_docids).intersection();
-            SetOperation::<DocumentId>::extend_collection(op, &mut counter);
-            count_map.insert(value.to_string(), counter.0);
-        }
-        facets_counts.insert(key, count_map);
-    }
-    facets_counts
-}

meilisearch-core/src/criterion/attribute.rs

@@ -1,37 +0,0 @@
-use std::cmp::Ordering;
-use slice_group_by::GroupBy;
-use crate::{RawDocument, MResult};
-use crate::bucket_sort::SimpleMatch;
-use super::{Criterion, Context, ContextMut, prepare_bare_matches};
-
-pub struct Attribute;
-
-impl Criterion for Attribute {
-    fn name(&self) -> &str { "attribute" }
-
-    fn prepare<'h, 'p, 'tag, 'txn, 'q, 'r>(
-        &self,
-        ctx: ContextMut<'h, 'p, 'tag, 'txn, 'q>,
-        documents: &mut [RawDocument<'r, 'tag>],
-    ) -> MResult<()>
-    {
-        prepare_bare_matches(documents, ctx.postings_lists, ctx.query_mapping);
-        Ok(())
-    }
-
-    fn evaluate(&self, _ctx: &Context, lhs: &RawDocument, rhs: &RawDocument) -> Ordering {
-        #[inline]
-        fn sum_of_attribute(matches: &[SimpleMatch]) -> usize {
-            let mut sum_of_attribute = 0;
-            for group in matches.linear_group_by_key(|bm| bm.query_index) {
-                sum_of_attribute += group[0].attribute as usize;
-            }
-            sum_of_attribute
-        }
-
-        let lhs = sum_of_attribute(&lhs.processed_matches);
-        let rhs = sum_of_attribute(&rhs.processed_matches);
-
-        lhs.cmp(&rhs)
-    }
-}

meilisearch-core/src/criterion/document_id.rs

@@ -1,16 +0,0 @@
-use std::cmp::Ordering;
-use crate::RawDocument;
-use super::{Criterion, Context};
-
-pub struct DocumentId;
-
-impl Criterion for DocumentId {
-    fn name(&self) -> &str { "stable document id" }
-
-    fn evaluate(&self, _ctx: &Context, lhs: &RawDocument, rhs: &RawDocument) -> Ordering {
-        let lhs = &lhs.id;
-        let rhs = &rhs.id;
-
-        lhs.cmp(rhs)
-    }
-}

meilisearch-core/src/criterion/exactness.rs

@@ -1,78 +0,0 @@
-use std::cmp::{Ordering, Reverse};
-use std::collections::hash_map::{HashMap, Entry};
-use meilisearch_schema::IndexedPos;
-use slice_group_by::GroupBy;
-use crate::{RawDocument, MResult};
-use crate::bucket_sort::BareMatch;
-use super::{Criterion, Context, ContextMut};
-
-pub struct Exactness;
-
-impl Criterion for Exactness {
-    fn name(&self) -> &str { "exactness" }
-
-    fn prepare<'h, 'p, 'tag, 'txn, 'q, 'r>(
-        &self,
-        ctx: ContextMut<'h, 'p, 'tag, 'txn, 'q>,
-        documents: &mut [RawDocument<'r, 'tag>],
-    ) -> MResult<()>
-    {
-        let store = ctx.documents_fields_counts_store;
-        let reader = ctx.reader;
-
-        'documents: for doc in documents {
-            doc.bare_matches.sort_unstable_by_key(|bm| (bm.query_index, Reverse(bm.is_exact)));
-
-            // mark the document if we find a "one word field" that matches
-            let mut fields_counts = HashMap::new();
-            for group in doc.bare_matches.linear_group_by_key(|bm| bm.query_index) {
-                for group in group.linear_group_by_key(|bm| bm.is_exact) {
-                    if !group[0].is_exact { break }
-
-                    for bm in group {
-                        for di in ctx.postings_lists[bm.postings_list].as_ref() {
-
-                            let attr = IndexedPos(di.attribute);
-                            let count = match fields_counts.entry(attr) {
-                                Entry::Occupied(entry) => *entry.get(),
-                                Entry::Vacant(entry) => {
-                                    let count = store.document_field_count(reader, doc.id, attr)?;
-                                    *entry.insert(count)
-                                },
-                            };
-
-                            if count == Some(1) {
-                                doc.contains_one_word_field = true;
-                                continue 'documents
-                            }
-                        }
-                    }
-                }
-            }
-        }
-
-        Ok(())
-    }
-
-    fn evaluate(&self, _ctx: &Context, lhs: &RawDocument, rhs: &RawDocument) -> Ordering {
-        #[inline]
-        fn sum_exact_query_words(matches: &[BareMatch]) -> usize {
-            let mut sum_exact_query_words = 0;
-
-            for group in matches.linear_group_by_key(|bm| bm.query_index) {
-                sum_exact_query_words += group[0].is_exact as usize;
-            }
-
-            sum_exact_query_words
-        }
-
-        // does it contains a "one word field"
-        lhs.contains_one_word_field.cmp(&rhs.contains_one_word_field).reverse()
-        // if not, with document contains the more exact words
-        .then_with(|| {
-            let lhs = sum_exact_query_words(&lhs.bare_matches);
-            let rhs = sum_exact_query_words(&rhs.bare_matches);
-            lhs.cmp(&rhs).reverse()
-        })
-    }
-}

meilisearch-core/src/criterion/mod.rs

@@ -1,292 +0,0 @@
-use std::cmp::{self, Ordering};
-use std::collections::HashMap;
-use std::ops::Range;
-
-use compact_arena::SmallArena;
-use sdset::SetBuf;
-use slice_group_by::GroupBy;
-
-use crate::bucket_sort::{SimpleMatch, PostingsListView};
-use crate::database::MainT;
-use crate::query_tree::QueryId;
-use crate::{store, RawDocument, MResult};
-
-mod typo;
-mod words;
-mod proximity;
-mod attribute;
-mod words_position;
-mod exactness;
-mod document_id;
-mod sort_by_attr;
-
-pub use self::typo::Typo;
-pub use self::words::Words;
-pub use self::proximity::Proximity;
-pub use self::attribute::Attribute;
-pub use self::words_position::WordsPosition;
-pub use self::exactness::Exactness;
-pub use self::document_id::DocumentId;
-pub use self::sort_by_attr::SortByAttr;
-
-pub trait Criterion {
-    fn name(&self) -> &str;
-
-    fn prepare<'h, 'p, 'tag, 'txn, 'q, 'r>(
-        &self,
-        _ctx: ContextMut<'h, 'p, 'tag, 'txn, 'q>,
-        _documents: &mut [RawDocument<'r, 'tag>],
-    ) -> MResult<()>
-    {
-        Ok(())
-    }
-
-    fn evaluate<'p, 'tag, 'txn, 'q, 'r>(
-        &self,
-        ctx: &Context<'p, 'tag, 'txn, 'q>,
-        lhs: &RawDocument<'r, 'tag>,
-        rhs: &RawDocument<'r, 'tag>,
-    ) -> Ordering;
-
-    #[inline]
-    fn eq<'p, 'tag, 'txn, 'q, 'r>(
-        &self,
-        ctx: &Context<'p, 'tag, 'txn, 'q>,
-        lhs: &RawDocument<'r, 'tag>,
-        rhs: &RawDocument<'r, 'tag>,
-    ) -> bool
-    {
-        self.evaluate(ctx, lhs, rhs) == Ordering::Equal
-    }
-}
-
-pub struct ContextMut<'h, 'p, 'tag, 'txn, 'q> {
-    pub reader: &'h heed::RoTxn<MainT>,
-    pub postings_lists: &'p mut SmallArena<'tag, PostingsListView<'txn>>,
-    pub query_mapping: &'q HashMap<QueryId, Range<usize>>,
-    pub documents_fields_counts_store: store::DocumentsFieldsCounts,
-}
-
-pub struct Context<'p, 'tag, 'txn, 'q> {
-    pub postings_lists: &'p SmallArena<'tag, PostingsListView<'txn>>,
-    pub query_mapping: &'q HashMap<QueryId, Range<usize>>,
-}
-
-#[derive(Default)]
-pub struct CriteriaBuilder<'a> {
-    inner: Vec<Box<dyn Criterion + 'a>>,
-}
-
-impl<'a> CriteriaBuilder<'a> {
-    pub fn new() -> CriteriaBuilder<'a> {
-        CriteriaBuilder { inner: Vec::new() }
-    }
-
-    pub fn with_capacity(capacity: usize) -> CriteriaBuilder<'a> {
-        CriteriaBuilder {
-            inner: Vec::with_capacity(capacity),
-        }
-    }
-
-    pub fn reserve(&mut self, additional: usize) {
-        self.inner.reserve(additional)
-    }
-
-    #[allow(clippy::should_implement_trait)]
-    pub fn add<C: 'a>(mut self, criterion: C) -> CriteriaBuilder<'a>
-    where
-        C: Criterion,
-    {
-        self.push(criterion);
-        self
-    }
-
-    pub fn push<C: 'a>(&mut self, criterion: C)
-    where
-        C: Criterion,
-    {
-        self.inner.push(Box::new(criterion));
-    }
-
-    pub fn build(self) -> Criteria<'a> {
-        Criteria { inner: self.inner }
-    }
-}
-
-pub struct Criteria<'a> {
-    inner: Vec<Box<dyn Criterion + 'a>>,
-}
-
-impl<'a> Default for Criteria<'a> {
-    fn default() -> Self {
-        CriteriaBuilder::with_capacity(7)
-            .add(Typo)
-            .add(Words)
-            .add(Proximity)
-            .add(Attribute)
-            .add(WordsPosition)
-            .add(Exactness)
-            .add(DocumentId)
-            .build()
-    }
-}
-
-impl<'a> AsRef<[Box<dyn Criterion + 'a>]> for Criteria<'a> {
-    fn as_ref(&self) -> &[Box<dyn Criterion + 'a>] {
-        &self.inner
-    }
-}
-
-fn prepare_query_distances<'a, 'tag, 'txn>(
-    documents: &mut [RawDocument<'a, 'tag>],
-    query_mapping: &HashMap<QueryId, Range<usize>>,
-    postings_lists: &SmallArena<'tag, PostingsListView<'txn>>,
-) {
-    for document in documents {
-        if !document.processed_distances.is_empty() { continue }
-
-        let mut processed = Vec::new();
-        for m in document.bare_matches.iter() {
-            if postings_lists[m.postings_list].is_empty() { continue }
-
-            let range = query_mapping[&(m.query_index as usize)].clone();
-            let new_len = cmp::max(range.end as usize, processed.len());
-            processed.resize(new_len, None);
-
-            for index in range {
-                let index = index as usize;
-                processed[index] = match processed[index] {
-                    Some(distance) if distance > m.distance => Some(m.distance),
-                    Some(distance) => Some(distance),
-                    None => Some(m.distance),
-                };
-            }
-        }
-
-        document.processed_distances = processed;
-    }
-}
-
-fn prepare_bare_matches<'a, 'tag, 'txn>(
-    documents: &mut [RawDocument<'a, 'tag>],
-    postings_lists: &mut SmallArena<'tag, PostingsListView<'txn>>,
-    query_mapping: &HashMap<QueryId, Range<usize>>,
-) {
-    for document in documents {
-        if !document.processed_matches.is_empty() { continue }
-
-        let mut processed = Vec::new();
-        for m in document.bare_matches.iter() {
-            let postings_list = &postings_lists[m.postings_list];
-            processed.reserve(postings_list.len());
-            for di in postings_list.as_ref() {
-                let simple_match = SimpleMatch {
-                    query_index: m.query_index,
-                    distance: m.distance,
-                    attribute: di.attribute,
-                    word_index: di.word_index,
-                    is_exact: m.is_exact,
-                };
-                processed.push(simple_match);
-            }
-        }
-
-        let processed = multiword_rewrite_matches(&mut processed, query_mapping);
-        document.processed_matches = processed.into_vec();
-    }
-}
-
-fn multiword_rewrite_matches(
-    matches: &mut [SimpleMatch],
-    query_mapping: &HashMap<QueryId, Range<usize>>,
-) -> SetBuf<SimpleMatch>
-{
-    matches.sort_unstable_by_key(|m| (m.attribute, m.word_index));
-
-    let mut padded_matches = Vec::with_capacity(matches.len());
-
-    // let before_padding = Instant::now();
-    // for each attribute of each document
-    for same_document_attribute in matches.linear_group_by_key(|m| m.attribute) {
-        // padding will only be applied
-        // to word indices in the same attribute
-        let mut padding = 0;
-        let mut iter = same_document_attribute.linear_group_by_key(|m| m.word_index);
-
-        // for each match at the same position
-        // in this document attribute
-        while let Some(same_word_index) = iter.next() {
-            // find the biggest padding
-            let mut biggest = 0;
-            for match_ in same_word_index {
-                let mut replacement = query_mapping[&(match_.query_index as usize)].clone();
-                let replacement_len = replacement.len();
-                let nexts = iter.remainder().linear_group_by_key(|m| m.word_index);
-
-                if let Some(query_index) = replacement.next() {
-                    let word_index = match_.word_index + padding as u16;
-                    let match_ = SimpleMatch { query_index, word_index, ..*match_ };
-                    padded_matches.push(match_);
-                }
-
-                let mut found = false;
-
-                // look ahead and if there already is a match
-                // corresponding to this padding word, abort the padding
-                'padding: for (x, next_group) in nexts.enumerate() {
-                    for (i, query_index) in replacement.clone().enumerate().skip(x) {
-                        let word_index = match_.word_index + padding as u16 + (i + 1) as u16;
-                        let padmatch = SimpleMatch { query_index, word_index, ..*match_ };
-
-                        for nmatch_ in next_group {
-                            let mut rep = query_mapping[&(nmatch_.query_index as usize)].clone();
-                            let query_index = rep.next().unwrap();
-                            if query_index == padmatch.query_index {
-                                if !found {
-                                    // if we find a corresponding padding for the
-                                    // first time we must push preceding paddings
-                                    for (i, query_index) in replacement.clone().enumerate().take(i) {
-                                        let word_index = match_.word_index + padding as u16 + (i + 1) as u16;
-                                        let match_ = SimpleMatch { query_index, word_index, ..*match_ };
-                                        padded_matches.push(match_);
-                                        biggest = biggest.max(i + 1);
-                                    }
-                                }
-
-                                padded_matches.push(padmatch);
-                                found = true;
-                                continue 'padding;
-                            }
-                        }
-                    }
-
-                    // if we do not find a corresponding padding in the
-                    // next groups so stop here and pad what was found
-                    break;
-                }
-
-                if !found {
-                    // if no padding was found in the following matches
-                    // we must insert the entire padding
-                    for (i, query_index) in replacement.enumerate() {
-                        let word_index = match_.word_index + padding as u16 + (i + 1) as u16;
-                        let match_ = SimpleMatch { query_index, word_index, ..*match_ };
-                        padded_matches.push(match_);
-                    }
-
-                    biggest = biggest.max(replacement_len - 1);
-                }
-            }
-
-            padding += biggest;
-        }
-    }
-
-    // debug!("padding matches took {:.02?}", before_padding.elapsed());
-
-    // With this check we can see that the loop above takes something
-    // like 43% of the search time even when no rewrite is needed.
-    // assert_eq!(before_matches, padded_matches);
-
-    SetBuf::from_dirty(padded_matches)
-}

meilisearch-core/src/criterion/proximity.rs

@@ -1,68 +0,0 @@
-use std::cmp::{self, Ordering};
-use slice_group_by::GroupBy;
-use crate::bucket_sort::{SimpleMatch};
-use crate::{RawDocument, MResult};
-use super::{Criterion, Context, ContextMut, prepare_bare_matches};
-
-const MAX_DISTANCE: u16 = 8;
-
-pub struct Proximity;
-
-impl Criterion for Proximity {
-    fn name(&self) -> &str { "proximity" }
-
-    fn prepare<'h, 'p, 'tag, 'txn, 'q, 'r>(
-        &self,
-        ctx: ContextMut<'h, 'p, 'tag, 'txn, 'q>,
-        documents: &mut [RawDocument<'r, 'tag>],
-    ) -> MResult<()>
-    {
-        prepare_bare_matches(documents, ctx.postings_lists, ctx.query_mapping);
-        Ok(())
-    }
-
-    fn evaluate(&self, _ctx: &Context, lhs: &RawDocument, rhs: &RawDocument) -> Ordering {
-        fn index_proximity(lhs: u16, rhs: u16) -> u16 {
-            if lhs < rhs {
-                cmp::min(rhs - lhs, MAX_DISTANCE)
-            } else {
-                cmp::min(lhs - rhs, MAX_DISTANCE) + 1
-            }
-        }
-
-        fn attribute_proximity(lhs: SimpleMatch, rhs: SimpleMatch) -> u16 {
-            if lhs.attribute != rhs.attribute { MAX_DISTANCE }
-            else { index_proximity(lhs.word_index, rhs.word_index) }
-        }
-
-        fn min_proximity(lhs: &[SimpleMatch], rhs: &[SimpleMatch]) -> u16 {
-            let mut min_prox = u16::max_value();
-            for a in lhs {
-                for b in rhs {
-                    let prox = attribute_proximity(*a, *b);
-                    min_prox = cmp::min(min_prox, prox);
-                }
-            }
-            min_prox
-        }
-
-        fn matches_proximity(matches: &[SimpleMatch],) -> u16 {
-            let mut proximity = 0;
-            let mut iter = matches.linear_group_by_key(|m| m.query_index);
-
-            // iterate over groups by windows of size 2
-            let mut last = iter.next();
-            while let (Some(lhs), Some(rhs)) = (last, iter.next()) {
-                proximity += min_proximity(lhs, rhs);
-                last = Some(rhs);
-            }
-
-            proximity
-        }
-
-        let lhs = matches_proximity(&lhs.processed_matches);
-        let rhs = matches_proximity(&rhs.processed_matches);
-
-        lhs.cmp(&rhs)
-    }
-}

meilisearch-core/src/criterion/sort_by_attr.rs

@@ -1,129 +0,0 @@
-use std::cmp::Ordering;
-use std::error::Error;
-use std::fmt;
-use meilisearch_schema::{Schema, FieldId};
-use crate::{RankedMap, RawDocument};
-use super::{Criterion, Context};
-
-/// An helper struct that permit to sort documents by
-/// some of their stored attributes.
-///
-/// # Note
-///
-/// If a document cannot be deserialized it will be considered [`None`][].
-///
-/// Deserialized documents are compared like `Some(doc0).cmp(&Some(doc1))`,
-/// so you must check the [`Ord`] of `Option` implementation.
-///
-/// [`None`]: https://doc.rust-lang.org/std/option/enum.Option.html#variant.None
-/// [`Ord`]: https://doc.rust-lang.org/std/option/enum.Option.html#impl-Ord
-///
-/// # Example
-///
-/// ```ignore
-/// use serde_derive::Deserialize;
-/// use meilisearch::rank::criterion::*;
-///
-/// let custom_ranking = SortByAttr::lower_is_better(&ranked_map, &schema, "published_at")?;
-///
-/// let builder = CriteriaBuilder::with_capacity(8)
-///        .add(Typo)
-///        .add(Words)
-///        .add(Proximity)
-///        .add(Attribute)
-///        .add(WordsPosition)
-///        .add(Exactness)
-///        .add(custom_ranking)
-///        .add(DocumentId);
-///
-/// let criterion = builder.build();
-///
-/// ```
-pub struct SortByAttr<'a> {
-    ranked_map: &'a RankedMap,
-    field_id: FieldId,
-    reversed: bool,
-}
-
-impl<'a> SortByAttr<'a> {
-    pub fn lower_is_better(
-        ranked_map: &'a RankedMap,
-        schema: &Schema,
-        attr_name: &str,
-    ) -> Result<SortByAttr<'a>, SortByAttrError> {
-        SortByAttr::new(ranked_map, schema, attr_name, false)
-    }
-
-    pub fn higher_is_better(
-        ranked_map: &'a RankedMap,
-        schema: &Schema,
-        attr_name: &str,
-    ) -> Result<SortByAttr<'a>, SortByAttrError> {
-        SortByAttr::new(ranked_map, schema, attr_name, true)
-    }
-
-    fn new(
-        ranked_map: &'a RankedMap,
-        schema: &Schema,
-        attr_name: &str,
-        reversed: bool,
-    ) -> Result<SortByAttr<'a>, SortByAttrError> {
-        let field_id = match schema.id(attr_name) {
-            Some(field_id) => field_id,
-            None => return Err(SortByAttrError::AttributeNotFound),
-        };
-
-        if !schema.is_ranked(field_id) {
-            return Err(SortByAttrError::AttributeNotRegisteredForRanking);
-        }
-
-        Ok(SortByAttr {
-            ranked_map,
-            field_id,
-            reversed,
-        })
-    }
-}
-
-impl Criterion for SortByAttr<'_> {
-    fn name(&self) -> &str {
-        "sort by attribute"
-    }
-
-    fn evaluate(&self, _ctx: &Context, lhs: &RawDocument, rhs: &RawDocument) -> Ordering {
-        let lhs = self.ranked_map.get(lhs.id, self.field_id);
-        let rhs = self.ranked_map.get(rhs.id, self.field_id);
-
-        match (lhs, rhs) {
-            (Some(lhs), Some(rhs)) => {
-                let order = lhs.cmp(&rhs);
-                if self.reversed {
-                    order.reverse()
-                } else {
-                    order
-                }
-            }
-            (None, Some(_)) => Ordering::Greater,
-            (Some(_), None) => Ordering::Less,
-            (None, None) => Ordering::Equal,
-        }
-    }
-}
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
-pub enum SortByAttrError {
-    AttributeNotFound,
-    AttributeNotRegisteredForRanking,
-}
-
-impl fmt::Display for SortByAttrError {
-    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        use SortByAttrError::*;
-        match self {
-            AttributeNotFound => f.write_str("attribute not found in the schema"),
-            AttributeNotRegisteredForRanking => f.write_str("attribute not registered for ranking"),
-        }
-    }
-}
-
-impl Error for SortByAttrError {}

meilisearch-core/src/criterion/typo.rs

@@ -1,56 +0,0 @@
-use std::cmp::Ordering;
-use crate::{RawDocument, MResult};
-use super::{Criterion, Context, ContextMut, prepare_query_distances};
-
-pub struct Typo;
-
-impl Criterion for Typo {
-    fn name(&self) -> &str { "typo" }
-
-    fn prepare<'h, 'p, 'tag, 'txn, 'q, 'r>(
-        &self,
-        ctx: ContextMut<'h, 'p, 'tag, 'txn, 'q>,
-        documents: &mut [RawDocument<'r, 'tag>],
-    ) -> MResult<()>
-    {
-        prepare_query_distances(documents, ctx.query_mapping, ctx.postings_lists);
-        Ok(())
-    }
-
-    fn evaluate(&self, _ctx: &Context, lhs: &RawDocument, rhs: &RawDocument) -> Ordering {
-        // This function is a wrong logarithmic 10 function.
-        // It is safe to panic on input number higher than 3,
-        // the number of typos is never bigger than that.
-        #[inline]
-        #[allow(clippy::approx_constant)]
-        fn custom_log10(n: u8) -> f32 {
-            match n {
-                0 => 0.0,     // log(1)
-                1 => 0.30102, // log(2)
-                2 => 0.47712, // log(3)
-                3 => 0.60205, // log(4)
-                _ => panic!("invalid number"),
-            }
-        }
-
-        #[inline]
-        fn compute_typos(distances: &[Option<u8>]) -> usize {
-            let mut number_words: usize = 0;
-            let mut sum_typos = 0.0;
-
-            for distance in distances {
-                if let Some(distance) = distance {
-                    sum_typos += custom_log10(*distance);
-                    number_words += 1;
-                }
-            }
-
-            (number_words as f32 / (sum_typos + 1.0) * 1000.0) as usize
-        }
-
-        let lhs = compute_typos(&lhs.processed_distances);
-        let rhs = compute_typos(&rhs.processed_distances);
-
-        lhs.cmp(&rhs).reverse()
-    }
-}

meilisearch-core/src/criterion/words.rs

@@ -1,31 +0,0 @@
-use std::cmp::Ordering;
-use crate::{RawDocument, MResult};
-use super::{Criterion, Context, ContextMut, prepare_query_distances};
-
-pub struct Words;
-
-impl Criterion for Words {
-    fn name(&self) -> &str { "words" }
-
-    fn prepare<'h, 'p, 'tag, 'txn, 'q, 'r>(
-        &self,
-        ctx: ContextMut<'h, 'p, 'tag, 'txn, 'q>,
-        documents: &mut [RawDocument<'r, 'tag>],
-    ) -> MResult<()>
-    {
-        prepare_query_distances(documents, ctx.query_mapping, ctx.postings_lists);
-        Ok(())
-    }
-
-    fn evaluate(&self, _ctx: &Context, lhs: &RawDocument, rhs: &RawDocument) -> Ordering {
-        #[inline]
-        fn number_of_query_words(distances: &[Option<u8>]) -> usize {
-            distances.iter().cloned().filter(Option::is_some).count()
-        }
-
-        let lhs = number_of_query_words(&lhs.processed_distances);
-        let rhs = number_of_query_words(&rhs.processed_distances);
-
-        lhs.cmp(&rhs).reverse()
-    }
-}

meilisearch-core/src/criterion/words_position.rs

@@ -1,37 +0,0 @@
-use std::cmp::Ordering;
-use slice_group_by::GroupBy;
-use crate::bucket_sort::SimpleMatch;
-use crate::{RawDocument, MResult};
-use super::{Criterion, Context, ContextMut, prepare_bare_matches};
-
-pub struct WordsPosition;
-
-impl Criterion for WordsPosition {
-    fn name(&self) -> &str { "words position" }
-
-    fn prepare<'h, 'p, 'tag, 'txn, 'q, 'r>(
-        &self,
-        ctx: ContextMut<'h, 'p, 'tag, 'txn, 'q>,
-        documents: &mut [RawDocument<'r, 'tag>],
-    ) -> MResult<()>
-    {
-        prepare_bare_matches(documents, ctx.postings_lists, ctx.query_mapping);
-        Ok(())
-    }
-
-    fn evaluate(&self, _ctx: &Context, lhs: &RawDocument, rhs: &RawDocument) -> Ordering {
-        #[inline]
-        fn sum_words_position(matches: &[SimpleMatch]) -> usize {
-            let mut sum_words_position = 0;
-            for group in matches.linear_group_by_key(|bm| bm.query_index) {
-                sum_words_position += group[0].word_index as usize;
-            }
-            sum_words_position
-        }
-
-        let lhs = sum_words_position(&lhs.processed_matches);
-        let rhs = sum_words_position(&rhs.processed_matches);
-
-        lhs.cmp(&rhs)
-    }
-}

meilisearch-core/src/distinct_map.rs

@@ -1,103 +0,0 @@
-use hashbrown::HashMap;
-use std::hash::Hash;
-
-pub struct DistinctMap<K> {
-    inner: HashMap<K, usize>,
-    limit: usize,
-    len: usize,
-}
-
-impl<K: Hash + Eq> DistinctMap<K> {
-    pub fn new(limit: usize) -> Self {
-        DistinctMap {
-            inner: HashMap::new(),
-            limit,
-            len: 0,
-        }
-    }
-
-    pub fn len(&self) -> usize {
-        self.len
-    }
-}
-
-pub struct BufferedDistinctMap<'a, K> {
-    internal: &'a mut DistinctMap<K>,
-    inner: HashMap<K, usize>,
-    len: usize,
-}
-
-impl<'a, K: Hash + Eq> BufferedDistinctMap<'a, K> {
-    pub fn new(internal: &'a mut DistinctMap<K>) -> BufferedDistinctMap<'a, K> {
-        BufferedDistinctMap {
-            internal,
-            inner: HashMap::new(),
-            len: 0,
-        }
-    }
-
-    pub fn register(&mut self, key: K) -> bool {
-        let internal_seen = self.internal.inner.get(&key).unwrap_or(&0);
-        let inner_seen = self.inner.entry(key).or_insert(0);
-        let seen = *internal_seen + *inner_seen;
-
-        if seen < self.internal.limit {
-            *inner_seen += 1;
-            self.len += 1;
-            true
-        } else {
-            false
-        }
-    }
-
-    pub fn register_without_key(&mut self) -> bool {
-        self.len += 1;
-        true
-    }
-
-    pub fn transfert_to_internal(&mut self) {
-        for (k, v) in self.inner.drain() {
-            let value = self.internal.inner.entry(k).or_insert(0);
-            *value += v;
-        }
-
-        self.internal.len += self.len;
-        self.len = 0;
-    }
-
-    pub fn len(&self) -> usize {
-        self.internal.len() + self.len
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn easy_distinct_map() {
-        let mut map = DistinctMap::new(2);
-        let mut buffered = BufferedDistinctMap::new(&mut map);
-
-        for x in &[1, 1, 1, 2, 3, 4, 5, 6, 6, 6, 6, 6] {
-            buffered.register(x);
-        }
-        buffered.transfert_to_internal();
-        assert_eq!(map.len(), 8);
-
-        let mut map = DistinctMap::new(2);
-        let mut buffered = BufferedDistinctMap::new(&mut map);
-        assert_eq!(buffered.register(1), true);
-        assert_eq!(buffered.register(1), true);
-        assert_eq!(buffered.register(1), false);
-        assert_eq!(buffered.register(1), false);
-
-        assert_eq!(buffered.register(2), true);
-        assert_eq!(buffered.register(3), true);
-        assert_eq!(buffered.register(2), true);
-        assert_eq!(buffered.register(2), false);
-
-        buffered.transfert_to_internal();
-        assert_eq!(map.len(), 5);
-    }
-}

meilisearch-core/src/error.rs

@@ -1,224 +0,0 @@
-use crate::serde::{DeserializerError, SerializerError};
-use serde_json::Error as SerdeJsonError;
-use pest::error::Error as PestError;
-use crate::filters::Rule;
-use std::{error, fmt, io};
-
-pub use bincode::Error as BincodeError;
-pub use fst::Error as FstError;
-pub use heed::Error as HeedError;
-pub use pest::error as pest_error;
-
-use meilisearch_error::{ErrorCode, Code};
-
-pub type MResult<T> = Result<T, Error>;
-
-#[derive(Debug)]
-pub enum Error {
-    Bincode(bincode::Error),
-    Deserializer(DeserializerError),
-    FacetError(FacetError),
-    FilterParseError(PestError<Rule>),
-    Fst(fst::Error),
-    Heed(heed::Error),
-    IndexAlreadyExists,
-    Io(io::Error),
-    MaxFieldsLimitExceeded,
-    MissingDocumentId,
-    MissingPrimaryKey,
-    Schema(meilisearch_schema::Error),
-    SchemaMissing,
-    SerdeJson(SerdeJsonError),
-    Serializer(SerializerError),
-    VersionMismatch(String),
-    WordIndexMissing,
-}
-
-impl ErrorCode for Error {
-    fn error_code(&self) -> Code {
-        use Error::*;
-
-        match self {
-            FacetError(_) => Code::Facet,
-            FilterParseError(_) => Code::Filter,
-            IndexAlreadyExists => Code::IndexAlreadyExists,
-            MissingPrimaryKey => Code::MissingPrimaryKey,
-            MissingDocumentId => Code::MissingDocumentId,
-            MaxFieldsLimitExceeded => Code::MaxFieldsLimitExceeded,
-            Schema(s) =>  s.error_code(),
-            WordIndexMissing
-            | SchemaMissing => Code::InvalidState,
-            Heed(_)
-            | Fst(_)
-            | SerdeJson(_)
-            | Bincode(_)
-            | Serializer(_)
-            | Deserializer(_)
-            | VersionMismatch(_)
-            | Io(_) => Code::Internal,
-        }
-    }
-}
-
-impl From<io::Error> for Error {
-    fn from(error: io::Error) -> Error {
-        Error::Io(error)
-    }
-}
-
-impl From<PestError<Rule>> for Error {
-    fn from(error: PestError<Rule>) -> Error {
-        Error::FilterParseError(error.renamed_rules(|r| {
-            let s  = match r {
-                Rule::or => "OR",
-                Rule::and => "AND",
-                Rule::not => "NOT",
-                Rule::string => "string",
-                Rule::word => "word",
-                Rule::greater => "field > value",
-                Rule::less => "field < value",
-                Rule::eq => "field = value",
-                Rule::leq => "field <= value",
-                Rule::geq => "field >= value",
-                Rule::key => "key",
-                _ => "other",
-            };
-            s.to_string()
-        }))
-    }
-}
-
-impl From<FacetError> for Error {
-    fn from(error: FacetError) -> Error {
-        Error::FacetError(error)
-    }
-}
-
-impl From<meilisearch_schema::Error> for Error {
-    fn from(error: meilisearch_schema::Error) -> Error {
-        Error::Schema(error)
-    }
-}
-
-impl From<HeedError> for Error {
-    fn from(error: HeedError) -> Error {
-        Error::Heed(error)
-    }
-}
-
-impl From<FstError> for Error {
-    fn from(error: FstError) -> Error {
-        Error::Fst(error)
-    }
-}
-
-impl From<SerdeJsonError> for Error {
-    fn from(error: SerdeJsonError) -> Error {
-        Error::SerdeJson(error)
-    }
-}
-
-impl From<BincodeError> for Error {
-    fn from(error: BincodeError) -> Error {
-        Error::Bincode(error)
-    }
-}
-
-impl From<SerializerError> for Error {
-    fn from(error: SerializerError) -> Error {
-        match error {
-            SerializerError::DocumentIdNotFound => Error::MissingDocumentId,
-            e => Error::Serializer(e),
-        }
-    }
-}
-
-impl From<DeserializerError> for Error {
-    fn from(error: DeserializerError) -> Error {
-        Error::Deserializer(error)
-    }
-}
-
-impl fmt::Display for Error {
-    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        use self::Error::*;
-        match self {
-            Bincode(e) => write!(f, "bincode error; {}", e),
-            Deserializer(e) => write!(f, "deserializer error; {}", e),
-            FacetError(e) => write!(f, "error processing facet filter: {}", e),
-            FilterParseError(e) => write!(f, "error parsing filter; {}", e),
-            Fst(e) => write!(f, "fst error; {}", e),
-            Heed(e) => write!(f, "heed error; {}", e),
-            IndexAlreadyExists => write!(f, "index already exists"),
-            Io(e) => write!(f, "{}", e),
-            MaxFieldsLimitExceeded => write!(f, "maximum number of fields in a document exceeded"),
-            MissingDocumentId => write!(f, "document id is missing"),
-            MissingPrimaryKey => write!(f, "schema cannot be built without a primary key"),
-            Schema(e) => write!(f, "schema error; {}", e),
-            SchemaMissing => write!(f, "this index does not have a schema"),
-            SerdeJson(e) => write!(f, "serde json error; {}", e),
-            Serializer(e) => write!(f, "serializer error; {}", e),
-            VersionMismatch(version) => write!(f, "Cannot open database, expected MeiliSearch engine version: {}, current engine version: {}.{}.{}",
-                version,
-                env!("CARGO_PKG_VERSION_MAJOR"),
-                env!("CARGO_PKG_VERSION_MINOR"),
-                env!("CARGO_PKG_VERSION_PATCH")),
-            WordIndexMissing => write!(f, "this index does not have a word index"),
-        }
-    }
-}
-
-impl error::Error for Error {}
-
-struct FilterParseError(PestError<Rule>);
-
-impl fmt::Display for FilterParseError {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        use crate::pest_error::LineColLocation::*;
-
-        let (line, column) = match self.0.line_col {
-            Span((line, _), (column, _)) => (line, column),
-            Pos((line, column)) => (line, column),
-        };
-        write!(f, "parsing error on line {} at column {}: {}", line, column, self.0.variant.message())
-    }
-}
-
-#[derive(Debug)]
-pub enum FacetError {
-    EmptyArray,
-    ParsingError(String),
-    UnexpectedToken { expected: &'static [&'static str], found: String },
-    InvalidFormat(String),
-    AttributeNotFound(String),
-    AttributeNotSet { expected: Vec<String>, found: String },
-    InvalidDocumentAttribute(String),
-    NoAttributesForFaceting,
-}
-
-impl FacetError {
-    pub fn unexpected_token(expected: &'static [&'static str], found: impl ToString) -> FacetError {
-        FacetError::UnexpectedToken{ expected, found: found.to_string() }
-    }
-
-    pub fn attribute_not_set(expected: Vec<String>, found: impl ToString) -> FacetError {
-        FacetError::AttributeNotSet{ expected, found: found.to_string() }
-    }
-}
-
-impl fmt::Display for FacetError {
-    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        use FacetError::*;
-
-        match self {
-            EmptyArray => write!(f, "empty array in facet filter is unspecified behavior"),
-            ParsingError(msg) => write!(f, "parsing error: {}", msg),
-            UnexpectedToken { expected, found } => write!(f, "unexpected token {}, expected {}", found, expected.join("or")),
-            InvalidFormat(found) => write!(f, "invalid facet: {}, facets should be \"facetName:facetValue\"", found),
-            AttributeNotFound(attr) => write!(f, "unknown {:?} attribute", attr),
-            AttributeNotSet { found, expected } => write!(f, "`{}` is not set as a faceted attribute. available facet attributes: {}", found, expected.join(", ")),
-            InvalidDocumentAttribute(attr) => write!(f, "invalid document attribute {}, accepted types: String and [String]", attr),
-            NoAttributesForFaceting => write!(f, "impossible to perform faceted search, no attributes for faceting are set"),
-        }
-    }
-}

meilisearch-core/src/facets.rs

@@ -1,357 +0,0 @@
-use std::borrow::Cow;
-use std::collections::HashMap;
-use std::hash::Hash;
-use std::ops::Deref;
-
-use cow_utils::CowUtils;
-use either::Either;
-use heed::types::{Str, OwnedType};
-use indexmap::IndexMap;
-use serde_json::Value;
-
-use meilisearch_schema::{FieldId, Schema};
-use meilisearch_types::DocumentId;
-
-use crate::database::MainT;
-use crate::error::{FacetError, MResult};
-use crate::store::BEU16;
-
-/// Data structure used to represent a boolean expression in the form of nested arrays.
-/// Values in the outer array are and-ed together, values in the inner arrays are or-ed together.
-#[derive(Debug, PartialEq)]
-pub struct FacetFilter(Vec<Either<Vec<FacetKey>, FacetKey>>);
-
-impl Deref for FacetFilter {
-    type Target = Vec<Either<Vec<FacetKey>, FacetKey>>;
-
-    fn deref(&self) -> &Self::Target {
-        &self.0
-    }
-}
-
-impl FacetFilter {
-    pub fn from_str(
-        s: &str,
-        schema: &Schema,
-        attributes_for_faceting: &[FieldId],
-    ) -> MResult<FacetFilter> {
-        if attributes_for_faceting.is_empty() {
-            return Err(FacetError::NoAttributesForFaceting.into());
-        }
-        let parsed = serde_json::from_str::<Value>(s).map_err(|e| FacetError::ParsingError(e.to_string()))?;
-        let mut filter = Vec::new();
-        match parsed {
-            Value::Array(and_exprs) => {
-                if and_exprs.is_empty() {
-                    return Err(FacetError::EmptyArray.into());
-                }
-                for expr in and_exprs {
-                    match expr {
-                        Value::String(s) => {
-                            let key = FacetKey::from_str( &s, schema, attributes_for_faceting)?;
-                            filter.push(Either::Right(key));
-                        }
-                        Value::Array(or_exprs) => {
-                            if or_exprs.is_empty() {
-                                return Err(FacetError::EmptyArray.into());
-                            }
-                            let mut inner = Vec::new();
-                            for expr in or_exprs {
-                                match expr {
-                                    Value::String(s) => {
-                                        let key = FacetKey::from_str( &s, schema, attributes_for_faceting)?;
-                                        inner.push(key);
-                                    }
-                                    bad_value => return Err(FacetError::unexpected_token(&["String"], bad_value).into()),
-                                }
-                            }
-                            filter.push(Either::Left(inner));
-                        }
-                        bad_value => return Err(FacetError::unexpected_token(&["Array", "String"], bad_value).into()),
-                    }
-                }
-                Ok(Self(filter))
-            }
-            bad_value => Err(FacetError::unexpected_token(&["Array"], bad_value).into()),
-        }
-    }
-}
-
-#[derive(Debug, Eq, PartialEq, Hash)]
-#[repr(C)]
-pub struct FacetKey(FieldId, String);
-
-impl FacetKey {
-    pub fn new(field_id: FieldId, value: String) -> Self {
-        let value = match value.cow_to_lowercase() {
-                Cow::Borrowed(_) => value,
-                Cow::Owned(s) => s,
-        };
-        Self(field_id, value)
-    }
-
-    pub fn key(&self) -> FieldId {
-        self.0
-    }
-
-    pub fn value(&self) -> &str {
-        &self.1
-    }
-
-    // TODO improve parser
-    fn from_str(
-        s: &str,
-        schema: &Schema,
-        attributes_for_faceting: &[FieldId],
-    ) -> Result<Self, FacetError> {
-        let mut split = s.splitn(2, ':');
-        let key = split
-            .next()
-            .ok_or_else(|| FacetError::InvalidFormat(s.to_string()))?
-            .trim();
-        let field_id = schema
-            .id(key)
-            .ok_or_else(|| FacetError::AttributeNotFound(key.to_string()))?;
-
-        if !attributes_for_faceting.contains(&field_id) {
-            return Err(FacetError::attribute_not_set(
-                    attributes_for_faceting
-                    .iter()
-                    .filter_map(|&id| schema.name(id))
-                    .map(str::to_string)
-                    .collect::<Vec<_>>(),
-                    key))
-        }
-        let value = split
-            .next()
-            .ok_or_else(|| FacetError::InvalidFormat(s.to_string()))?
-            .trim();
-        // unquoting the string if need be:
-        let mut indices = value.char_indices();
-        let value =  match (indices.next(), indices.last()) {
-            (Some((s, '\'')), Some((e, '\''))) |
-            (Some((s, '\"')), Some((e, '\"'))) => value[s + 1..e].to_string(),
-            _ => value.to_string(),
-        };
-        Ok(Self::new(field_id, value))
-    }
-}
-
-impl<'a> heed::BytesEncode<'a> for FacetKey {
-    type EItem = FacetKey;
-
-    fn bytes_encode(item: &'a Self::EItem) -> Option<Cow<'a, [u8]>> {
-        let mut buffer = Vec::with_capacity(2 + item.1.len());
-        let id = BEU16::new(item.key().into());
-        let id_bytes = OwnedType::bytes_encode(&id)?;
-        let value_bytes = Str::bytes_encode(item.value())?;
-        buffer.extend_from_slice(id_bytes.as_ref());
-        buffer.extend_from_slice(value_bytes.as_ref());
-        Some(Cow::Owned(buffer))
-    }
-}
-
-impl<'a> heed::BytesDecode<'a> for FacetKey {
-    type DItem = FacetKey;
-
-    fn bytes_decode(bytes: &'a [u8]) -> Option<Self::DItem> {
-        let (id_bytes, value_bytes) = bytes.split_at(2);
-        let id = OwnedType::<BEU16>::bytes_decode(id_bytes)?;
-        let id = id.get().into();
-        let string = Str::bytes_decode(&value_bytes)?;
-        Some(FacetKey(id, string.to_string()))
-    }
-}
-
-pub fn add_to_facet_map(
-    facet_map: &mut HashMap<FacetKey, (String, Vec<DocumentId>)>,
-    field_id: FieldId,
-    value: Value,
-    document_id: DocumentId,
-) -> Result<(), FacetError> {
-    let value = match value {
-        Value::String(s) => s,
-        // ignore null
-        Value::Null => return Ok(()),
-        value => return Err(FacetError::InvalidDocumentAttribute(value.to_string())),
-    };
-    let key = FacetKey::new(field_id, value.clone());
-    facet_map.entry(key).or_insert_with(|| (value, Vec::new())).1.push(document_id);
-    Ok(())
-}
-
-pub fn facet_map_from_docids(
-    rtxn: &heed::RoTxn<MainT>,
-    index: &crate::Index,
-    document_ids: &[DocumentId],
-    attributes_for_facetting: &[FieldId],
-) -> MResult<HashMap<FacetKey, (String, Vec<DocumentId>)>> {
-    // A hashmap that ascociate a facet key to a pair containing the original facet attribute
-    // string with it's case preserved, and a list of document ids for that facet attribute.
-    let mut facet_map: HashMap<FacetKey, (String, Vec<DocumentId>)> = HashMap::new();
-    for document_id in document_ids {
-        for result in index
-            .documents_fields
-            .document_fields(rtxn, *document_id)?
-        {
-            let (field_id, bytes) = result?;
-            if attributes_for_facetting.contains(&field_id) {
-                match serde_json::from_slice(bytes)? {
-                    Value::Array(values) => {
-                        for v in values {
-                            add_to_facet_map(&mut facet_map, field_id, v, *document_id)?;
-                        }
-                    }
-                    v => add_to_facet_map(&mut facet_map, field_id, v, *document_id)?,
-                };
-            }
-        }
-    }
-    Ok(facet_map)
-}
-
-pub fn facet_map_from_docs(
-    schema: &Schema,
-    documents: &HashMap<DocumentId, IndexMap<String, Value>>,
-    attributes_for_facetting: &[FieldId],
-) -> MResult<HashMap<FacetKey, (String, Vec<DocumentId>)>> {
-    let mut facet_map = HashMap::new();
-    let attributes_for_facetting = attributes_for_facetting
-        .iter()
-        .filter_map(|&id| schema.name(id).map(|name| (id, name)))
-        .collect::<Vec<_>>();
-
-    for (id, document) in documents {
-        for (field_id, name) in &attributes_for_facetting {
-            if let Some(value) = document.get(*name) {
-                match value {
-                    Value::Array(values) => {
-                        for v in values {
-                            add_to_facet_map(&mut facet_map, *field_id, v.clone(), *id)?;
-                        }
-                    }
-                    v => add_to_facet_map(&mut facet_map, *field_id, v.clone(), *id)?,
-                }
-            }
-        }
-    }
-    Ok(facet_map)
-}
-
-#[cfg(test)]
-mod test {
-    use super::*;
-    use meilisearch_schema::Schema;
-
-    #[test]
-    fn test_facet_key() {
-        let mut schema = Schema::new();
-        let id = schema.insert_and_index("hello").unwrap();
-        let facet_list = [schema.id("hello").unwrap()];
-        assert_eq!(
-            FacetKey::from_str("hello:12", &schema, &facet_list).unwrap(),
-            FacetKey::new(id, "12".to_string())
-        );
-        assert_eq!(
-            FacetKey::from_str("hello:\"foo bar\"", &schema, &facet_list).unwrap(),
-            FacetKey::new(id, "foo bar".to_string())
-        );
-        assert_eq!(
-            FacetKey::from_str("hello:'foo bar'", &schema, &facet_list).unwrap(),
-            FacetKey::new(id, "foo bar".to_string())
-        );
-        // weird case
-        assert_eq!(
-            FacetKey::from_str("hello:blabla:machin", &schema, &facet_list).unwrap(),
-            FacetKey::new(id, "blabla:machin".to_string())
-        );
-
-        assert_eq!(
-            FacetKey::from_str("hello:\"\"", &schema, &facet_list).unwrap(),
-            FacetKey::new(id, "".to_string())
-        );
-
-        assert_eq!(
-            FacetKey::from_str("hello:'", &schema, &facet_list).unwrap(),
-            FacetKey::new(id, "'".to_string())
-        );
-        assert_eq!(
-            FacetKey::from_str("hello:''", &schema, &facet_list).unwrap(),
-            FacetKey::new(id, "".to_string())
-        );
-        assert!(FacetKey::from_str("hello", &schema, &facet_list).is_err());
-        assert!(FacetKey::from_str("toto:12", &schema, &facet_list).is_err());
-    }
-
-    #[test]
-    fn test_parse_facet_array() {
-        use either::Either::{Left, Right};
-        let mut schema = Schema::new();
-        let _id = schema.insert_and_index("hello").unwrap();
-        let facet_list = [schema.id("hello").unwrap()];
-        assert_eq!(
-            FacetFilter::from_str("[[\"hello:12\"]]", &schema, &facet_list).unwrap(),
-            FacetFilter(vec![Left(vec![FacetKey(FieldId(0), "12".to_string())])])
-        );
-        assert_eq!(
-            FacetFilter::from_str("[\"hello:12\"]", &schema, &facet_list).unwrap(),
-            FacetFilter(vec![Right(FacetKey(FieldId(0), "12".to_string()))])
-        );
-        assert_eq!(
-            FacetFilter::from_str("[\"hello:12\", \"hello:13\"]", &schema, &facet_list).unwrap(),
-            FacetFilter(vec![
-                Right(FacetKey(FieldId(0), "12".to_string())),
-                Right(FacetKey(FieldId(0), "13".to_string()))
-            ])
-        );
-        assert_eq!(
-            FacetFilter::from_str("[[\"hello:12\", \"hello:13\"]]", &schema, &facet_list).unwrap(),
-            FacetFilter(vec![Left(vec![
-                FacetKey(FieldId(0), "12".to_string()),
-                FacetKey(FieldId(0), "13".to_string())
-            ])])
-        );
-        assert_eq!(
-            FacetFilter::from_str(
-                "[[\"hello:12\", \"hello:13\"], \"hello:14\"]",
-                &schema,
-                &facet_list
-            )
-            .unwrap(),
-            FacetFilter(vec![
-                Left(vec![
-                    FacetKey(FieldId(0), "12".to_string()),
-                    FacetKey(FieldId(0), "13".to_string())
-                ]),
-                Right(FacetKey(FieldId(0), "14".to_string()))
-            ])
-        );
-
-        // invalid array depths
-        assert!(FacetFilter::from_str(
-            "[[[\"hello:12\", \"hello:13\"], \"hello:14\"]]",
-            &schema,
-            &facet_list
-        )
-        .is_err());
-        assert!(FacetFilter::from_str(
-            "[[[\"hello:12\", \"hello:13\"]], \"hello:14\"]]",
-            &schema,
-            &facet_list
-        )
-        .is_err());
-        assert!(FacetFilter::from_str("\"hello:14\"", &schema, &facet_list).is_err());
-
-        // unexisting key
-        assert!(FacetFilter::from_str("[\"foo:12\"]", &schema, &facet_list).is_err());
-
-        // invalid facet key
-        assert!(FacetFilter::from_str("[\"foo=12\"]", &schema, &facet_list).is_err());
-        assert!(FacetFilter::from_str("[\"foo12\"]", &schema, &facet_list).is_err());
-        assert!(FacetFilter::from_str("[\"\"]", &schema, &facet_list).is_err());
-
-        // empty array error
-        assert!(FacetFilter::from_str("[]", &schema, &facet_list).is_err());
-        assert!(FacetFilter::from_str("[\"hello:12\", []]", &schema, &facet_list).is_err());
-    }
-}

meilisearch-core/src/filters/condition.rs

@@ -1,276 +0,0 @@
-use std::str::FromStr;
-use std::cmp::Ordering;
-
-use crate::error::Error;
-use crate::{store::Index, DocumentId, MainT};
-use heed::RoTxn;
-use meilisearch_schema::{FieldId, Schema};
-use pest::error::{Error as PestError, ErrorVariant};
-use pest::iterators::Pair;
-use serde_json::{Value, Number};
-use super::parser::Rule;
-
-#[derive(Debug, PartialEq)]
-enum ConditionType {
-    Greater,
-    Less,
-    Equal,
-    LessEqual,
-    GreaterEqual,
-    NotEqual,
-}
-
-/// We need to infer type when the filter is constructed
-/// and match every possible types it can be parsed into.
-#[derive(Debug)]
-struct ConditionValue<'a> {
-    string: &'a str,
-    boolean: Option<bool>,
-    number: Option<Number>
-}
-
-impl<'a> ConditionValue<'a> {
-    pub fn new(value: &Pair<'a, Rule>) -> Self {
-        match value.as_rule() {
-            Rule::string | Rule::word => {
-                let string =  value.as_str();
-                let boolean = match value.as_str() {
-                    "true" => Some(true),
-                    "false" => Some(false),
-                    _ => None,
-                };
-                let number = Number::from_str(value.as_str()).ok();
-                ConditionValue { string, boolean, number }
-            },
-            _ => unreachable!(),
-        }
-    }
-
-    pub fn as_str(&self) -> &str {
-        self.string
-    }
-
-    pub fn as_number(&self) -> Option<&Number> {
-        self.number.as_ref()
-    }
-
-    pub fn as_bool(&self) -> Option<bool> {
-        self.boolean
-    }
-}
-
-#[derive(Debug)]
-pub struct Condition<'a> {
-    field: FieldId,
-    condition: ConditionType,
-    value: ConditionValue<'a>
-}
-
-fn get_field_value<'a>(schema: &Schema, pair: Pair<'a, Rule>) -> Result<(FieldId, ConditionValue<'a>), Error> {
-    let mut items = pair.into_inner();
-    // lexing ensures that we at least have a key
-    let key = items.next().unwrap();
-    let field = schema
-        .id(key.as_str())
-        .ok_or_else(|| PestError::new_from_span(
-                ErrorVariant::CustomError {
-                    message: format!(
-                                 "attribute `{}` not found, available attributes are: {}",
-                                 key.as_str(),
-                                 schema.names().collect::<Vec<_>>().join(", ")
-                             ),
-                },
-                key.as_span()))?;
-    let value = ConditionValue::new(&items.next().unwrap());
-    Ok((field, value))
-}
-
-// undefined behavior with big numbers
-fn compare_numbers(lhs: &Number, rhs: &Number) -> Option<Ordering> {
-    match (lhs.as_i64(), lhs.as_u64(), lhs.as_f64(),
-        rhs.as_i64(), rhs.as_u64(), rhs.as_f64()) {
-    //    i64   u64  f64  i64  u64  f64
-        (Some(lhs), _, _, Some(rhs), _, _) => lhs.partial_cmp(&rhs),
-        (_, Some(lhs), _, _, Some(rhs), _) => lhs.partial_cmp(&rhs),
-        (_, _, Some(lhs), _, _, Some(rhs)) => lhs.partial_cmp(&rhs),
-        (_, _, _, _, _, _) => None,
-    }
-}
-
-impl<'a> Condition<'a> {
-    pub fn less(
-        item: Pair<'a, Rule>,
-        schema: &'a Schema,
-    ) -> Result<Self, Error> {
-        let (field, value) = get_field_value(schema, item)?;
-        let condition = ConditionType::Less;
-        Ok(Self { field, condition, value })
-    }
-
-    pub fn greater(
-        item: Pair<'a, Rule>,
-        schema: &'a Schema,
-    ) -> Result<Self, Error> {
-        let (field, value) = get_field_value(schema, item)?;
-        let condition = ConditionType::Greater;
-        Ok(Self { field, condition, value })
-    }
-
-    pub fn neq(
-        item: Pair<'a, Rule>,
-        schema: &'a Schema,
-    ) -> Result<Self, Error> {
-        let (field, value) = get_field_value(schema, item)?;
-        let condition = ConditionType::NotEqual;
-        Ok(Self { field, condition, value })
-    }
-
-    pub fn geq(
-        item: Pair<'a, Rule>,
-        schema: &'a Schema,
-    ) -> Result<Self, Error> {
-        let (field, value) = get_field_value(schema, item)?;
-        let condition = ConditionType::GreaterEqual;
-        Ok(Self { field, condition, value })
-    }
-
-    pub fn leq(
-        item: Pair<'a, Rule>,
-        schema: &'a Schema,
-    ) -> Result<Self, Error> {
-        let (field, value) = get_field_value(schema, item)?;
-        let condition = ConditionType::LessEqual;
-        Ok(Self { field, condition, value })
-    }
-
-    pub fn eq(
-        item: Pair<'a, Rule>,
-        schema: &'a Schema,
-    ) -> Result<Self, Error> {
-        let (field, value) = get_field_value(schema, item)?;
-        let condition = ConditionType::Equal;
-        Ok(Self { field, condition, value })
-    }
-
-    pub fn test(
-        &self,
-        reader: &RoTxn<MainT>,
-        index: &Index,
-        document_id: DocumentId,
-    ) -> Result<bool, Error> {
-        match index.document_attribute::<Value>(reader, document_id, self.field)? {
-            Some(Value::Array(values)) => Ok(values.iter().any(|v| self.match_value(Some(v)))),
-            other => Ok(self.match_value(other.as_ref())),
-        }
-    }
-
-    fn match_value(&self, value: Option<&Value>) -> bool {
-        match value {
-            Some(Value::String(s)) => {
-                let value = self.value.as_str();
-                match self.condition {
-                    ConditionType::Equal => unicase::eq(value, &s),
-                    ConditionType::NotEqual => !unicase::eq(value, &s),
-                    _ => false
-                }
-            },
-            Some(Value::Number(n)) => { 
-                if let Some(value) = self.value.as_number() {
-                    if let Some(ord) = compare_numbers(&n, value) {
-                        let res =  match self.condition {
-                            ConditionType::Equal => ord == Ordering::Equal,
-                            ConditionType::NotEqual => ord != Ordering::Equal,
-                            ConditionType::GreaterEqual => ord != Ordering::Less,
-                            ConditionType::LessEqual => ord != Ordering::Greater,
-                            ConditionType::Greater => ord == Ordering::Greater,
-                            ConditionType::Less => ord == Ordering::Less,
-                        };
-                        return res
-                    } 
-                } 
-                false
-            },
-            Some(Value::Bool(b)) => {
-                if let Some(value) = self.value.as_bool() {
-                    let res = match self.condition {
-                        ConditionType::Equal => *b == value,
-                        ConditionType::NotEqual => *b != value,
-                        _ => false
-                    };
-                    return res
-                }
-                false
-            },
-            // if field is not supported (or not found), all values are different from it,
-            // so != should always return true in this case.
-            _ => self.condition == ConditionType::NotEqual,
-        }
-    }
-}
-
-#[cfg(test)]
-mod test {
-    use super::*;
-    use serde_json::Number;
-    use std::cmp::Ordering;
-
-    #[test]
-    fn test_number_comp() {
-        // test both u64
-        let n1 = Number::from(1u64);
-        let n2 = Number::from(2u64);
-        assert_eq!(Some(Ordering::Less), compare_numbers(&n1, &n2));
-        assert_eq!(Some(Ordering::Greater), compare_numbers(&n2, &n1));
-        let n1 = Number::from(1u64);
-        let n2 = Number::from(1u64);
-        assert_eq!(Some(Ordering::Equal), compare_numbers(&n1, &n2));
-
-        // test both i64
-        let n1 = Number::from(1i64);
-        let n2 = Number::from(2i64);
-        assert_eq!(Some(Ordering::Less), compare_numbers(&n1, &n2));
-        assert_eq!(Some(Ordering::Greater), compare_numbers(&n2, &n1));
-        let n1 = Number::from(1i64);
-        let n2 = Number::from(1i64);
-        assert_eq!(Some(Ordering::Equal), compare_numbers(&n1, &n2));
-
-        // test both f64
-        let n1 = Number::from_f64(1f64).unwrap();
-        let n2 = Number::from_f64(2f64).unwrap();
-        assert_eq!(Some(Ordering::Less), compare_numbers(&n1, &n2));
-        assert_eq!(Some(Ordering::Greater), compare_numbers(&n2, &n1));
-        let n1 = Number::from_f64(1f64).unwrap();
-        let n2 = Number::from_f64(1f64).unwrap();
-        assert_eq!(Some(Ordering::Equal), compare_numbers(&n1, &n2));
-
-        // test one u64 and one f64
-        let n1 = Number::from_f64(1f64).unwrap();
-        let n2 = Number::from(2u64);
-        assert_eq!(Some(Ordering::Less), compare_numbers(&n1, &n2));
-        assert_eq!(Some(Ordering::Greater), compare_numbers(&n2, &n1));
-
-        // equality
-        let n1 = Number::from_f64(1f64).unwrap();
-        let n2 = Number::from(1u64);
-        assert_eq!(Some(Ordering::Equal), compare_numbers(&n1, &n2));
-        assert_eq!(Some(Ordering::Equal), compare_numbers(&n2, &n1));
-
-        // float is neg
-        let n1 = Number::from_f64(-1f64).unwrap();
-        let n2 = Number::from(1u64);
-        assert_eq!(Some(Ordering::Less), compare_numbers(&n1, &n2));
-        assert_eq!(Some(Ordering::Greater), compare_numbers(&n2, &n1));
-
-        // float is too big
-        let n1 = Number::from_f64(std::f64::MAX).unwrap();
-        let n2 = Number::from(1u64);
-        assert_eq!(Some(Ordering::Greater), compare_numbers(&n1, &n2));
-        assert_eq!(Some(Ordering::Less), compare_numbers(&n2, &n1));
-
-        // misc
-        let n1 = Number::from_f64(std::f64::MAX).unwrap();
-        let n2 = Number::from(std::u64::MAX);
-        assert_eq!(Some(Ordering::Greater), compare_numbers(&n1, &n2));
-        assert_eq!(Some( Ordering::Less ), compare_numbers(&n2, &n1));
-    }
-}

meilisearch-core/src/filters/mod.rs

@@ -1,127 +0,0 @@
-mod parser;
-mod condition;
-
-pub(crate) use parser::Rule;
-
-use std::ops::Not;
-
-use condition::Condition;
-use crate::error::Error;
-use crate::{DocumentId, MainT, store::Index};
-use heed::RoTxn;
-use meilisearch_schema::Schema;
-use parser::{PREC_CLIMBER, FilterParser};
-use pest::iterators::{Pair, Pairs};
-use pest::Parser;
-
-type FilterResult<'a> = Result<Filter<'a>, Error>;
-
-#[derive(Debug)]
-pub enum Filter<'a> {
-    Condition(Condition<'a>),
-    Or(Box<Self>, Box<Self>),
-    And(Box<Self>, Box<Self>),
-    Not(Box<Self>),
-}
-
-impl<'a> Filter<'a> {
-    pub fn parse(expr: &'a str, schema: &'a Schema) -> FilterResult<'a> {
-        let mut lexed = FilterParser::parse(Rule::prgm, expr)?;
-        Self::build(lexed.next().unwrap().into_inner(), schema)
-    }
-
-    pub fn test(
-        &self,
-        reader: &RoTxn<MainT>,
-        index: &Index,
-        document_id: DocumentId,
-    ) -> Result<bool, Error> {
-        use Filter::*;
-        match self {
-            Condition(c) => c.test(reader, index, document_id),
-            Or(lhs, rhs) => Ok(
-                lhs.test(reader, index, document_id)? || rhs.test(reader, index, document_id)?
-            ),
-            And(lhs, rhs) => Ok(
-                lhs.test(reader, index, document_id)? && rhs.test(reader, index, document_id)?
-            ),
-            Not(op) => op.test(reader, index, document_id).map(bool::not),
-        }
-    }
-
-    fn build(expression: Pairs<'a, Rule>, schema: &'a Schema) -> FilterResult<'a> {
-        PREC_CLIMBER.climb(
-            expression,
-            |pair: Pair<Rule>| match pair.as_rule() {
-                Rule::eq => Ok(Filter::Condition(Condition::eq(pair, schema)?)),
-                Rule::greater => Ok(Filter::Condition(Condition::greater(pair, schema)?)),
-                Rule::less => Ok(Filter::Condition(Condition::less(pair, schema)?)),
-                Rule::neq => Ok(Filter::Condition(Condition::neq(pair, schema)?)),
-                Rule::geq => Ok(Filter::Condition(Condition::geq(pair, schema)?)),
-                Rule::leq => Ok(Filter::Condition(Condition::leq(pair, schema)?)),
-                Rule::prgm => Self::build(pair.into_inner(), schema),
-                Rule::term => Self::build(pair.into_inner(), schema),
-                Rule::not => Ok(Filter::Not(Box::new(Self::build(
-                    pair.into_inner(),
-                    schema,
-                )?))),
-                _ => unreachable!(),
-            },
-            |lhs: FilterResult, op: Pair<Rule>, rhs: FilterResult| match op.as_rule() {
-                Rule::or => Ok(Filter::Or(Box::new(lhs?), Box::new(rhs?))),
-                Rule::and => Ok(Filter::And(Box::new(lhs?), Box::new(rhs?))),
-                _ => unreachable!(),
-            },
-        )
-    }
-}
-
-#[cfg(test)]
-mod test {
-    use super::*;
-
-    #[test]
-    fn invalid_syntax() {
-        assert!(FilterParser::parse(Rule::prgm, "field : id").is_err());
-        assert!(FilterParser::parse(Rule::prgm, "field=hello hello").is_err());
-        assert!(FilterParser::parse(Rule::prgm, "field=hello OR OR").is_err());
-        assert!(FilterParser::parse(Rule::prgm, "OR field:hello").is_err());
-        assert!(FilterParser::parse(Rule::prgm, r#"field="hello world"#).is_err());
-        assert!(FilterParser::parse(Rule::prgm, r#"field='hello world"#).is_err());
-        assert!(FilterParser::parse(Rule::prgm, "NOT field=").is_err());
-        assert!(FilterParser::parse(Rule::prgm, "N").is_err());
-        assert!(FilterParser::parse(Rule::prgm, "(field=1").is_err());
-        assert!(FilterParser::parse(Rule::prgm, "(field=1))").is_err());
-        assert!(FilterParser::parse(Rule::prgm, "field=1ORfield=2").is_err());
-        assert!(FilterParser::parse(Rule::prgm, "field=1 ( OR field=2)").is_err());
-        assert!(FilterParser::parse(Rule::prgm, "hello world=1").is_err());
-        assert!(FilterParser::parse(Rule::prgm, "").is_err());
-        assert!(FilterParser::parse(Rule::prgm, r#"((((((hello=world)))))"#).is_err());
-    }
-
-    #[test]
-    fn valid_syntax() {
-        assert!(FilterParser::parse(Rule::prgm, "field = id").is_ok());
-        assert!(FilterParser::parse(Rule::prgm, "field=id").is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field >= 10"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field <= 10"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field="hello world""#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field='hello world'"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field > 10"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field < 10"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field < 10 AND NOT field=5"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field < 10 AND NOT field > 7.5"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field=true OR NOT field=5"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"NOT field=true OR NOT field=5"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field='hello world' OR ( NOT field=true OR NOT field=5 )"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field='hello \'worl\'d' OR ( NOT field=true OR NOT field=5 )"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"field="hello \"worl\"d" OR ( NOT field=true OR NOT field=5 )"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"((((((hello=world))))))"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#""foo bar" > 10"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#""foo bar" = 10"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"'foo bar' = 10"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"'foo bar' <= 10"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"'foo bar' != 10"#).is_ok());
-        assert!(FilterParser::parse(Rule::prgm, r#"bar != 10"#).is_ok());
-    }
-}

meilisearch-core/src/filters/parser/grammar.pest

@@ -1,28 +0,0 @@
-key = _{quoted | word}
-value = _{quoted | word}
-quoted = _{ (PUSH("'") | PUSH("\"")) ~ string ~ POP  }
-string = {char*}
-word = ${(LETTER | NUMBER | "_" | "-" | ".")+}
-
-char =  _{ !(PEEK | "\\") ~ ANY
-    | "\\" ~ (PEEK | "\\" | "/" | "b" | "f" | "n" | "r" | "t")
-    | "\\" ~ ("u" ~ ASCII_HEX_DIGIT{4})}
-
-condition = _{eq | greater | less | geq | leq | neq}
-geq = {key ~ ">=" ~ value}
-leq = {key ~ "<=" ~ value}
-neq = {key ~ "!=" ~ value}
-eq = {key ~ "=" ~ value}
-greater = {key ~ ">" ~ value}
-less = {key ~ "<" ~ value}
-
-prgm = {SOI ~ expr ~ EOI}
-expr = _{ ( term ~ (operation ~ term)* ) }
-term = { ("(" ~ expr ~ ")") | condition | not }
-operation = _{ and | or }
-	and = {"AND"}
-	or = {"OR"}
-
-not = {"NOT" ~ term}
-
-WHITESPACE = _{ " " }

meilisearch-core/src/filters/parser/mod.rs

@@ -1,12 +0,0 @@
-use once_cell::sync::Lazy;
-use pest::prec_climber::{Operator, Assoc, PrecClimber};
-
-pub static PREC_CLIMBER: Lazy<PrecClimber<Rule>> = Lazy::new(|| {
-    use Assoc::*;
-    use Rule::*;
-    pest::prec_climber::PrecClimber::new(vec![Operator::new(or, Left), Operator::new(and, Left)])
-});
-
-#[derive(Parser)]
-#[grammar = "filters/parser/grammar.pest"]
-pub struct FilterParser;

meilisearch-core/src/levenshtein.rs

@@ -1,134 +0,0 @@
-use std::cmp::min;
-use std::collections::BTreeMap;
-use std::ops::{Index, IndexMut};
-
-// A simple wrapper around vec so we can get contiguous but index it like it's 2D array.
-struct N2Array<T> {
-    y_size: usize,
-    buf: Vec<T>,
-}
-
-impl<T: Clone> N2Array<T> {
-    fn new(x: usize, y: usize, value: T) -> N2Array<T> {
-        N2Array {
-            y_size: y,
-            buf: vec![value; x * y],
-        }
-    }
-}
-
-impl<T> Index<(usize, usize)> for N2Array<T> {
-    type Output = T;
-
-    #[inline]
-    fn index(&self, (x, y): (usize, usize)) -> &T {
-        &self.buf[(x * self.y_size) + y]
-    }
-}
-
-impl<T> IndexMut<(usize, usize)> for N2Array<T> {
-    #[inline]
-    fn index_mut(&mut self, (x, y): (usize, usize)) -> &mut T {
-        &mut self.buf[(x * self.y_size) + y]
-    }
-}
-
-pub fn prefix_damerau_levenshtein(source: &[u8], target: &[u8]) -> (u32, usize) {
-    let (n, m) = (source.len(), target.len());
-
-    assert!(
-        n <= m,
-        "the source string must be shorter than the target one"
-    );
-
-    if n == 0 {
-        return (m as u32, 0);
-    }
-    if m == 0 {
-        return (n as u32, 0);
-    }
-
-    if n == m && source == target {
-        return (0, m);
-    }
-
-    let inf = n + m;
-    let mut matrix = N2Array::new(n + 2, m + 2, 0);
-
-    matrix[(0, 0)] = inf;
-    for i in 0..n + 1 {
-        matrix[(i + 1, 0)] = inf;
-        matrix[(i + 1, 1)] = i;
-    }
-    for j in 0..m + 1 {
-        matrix[(0, j + 1)] = inf;
-        matrix[(1, j + 1)] = j;
-    }
-
-    let mut last_row = BTreeMap::new();
-
-    for (row, char_s) in source.iter().enumerate() {
-        let mut last_match_col = 0;
-        let row = row + 1;
-
-        for (col, char_t) in target.iter().enumerate() {
-            let col = col + 1;
-            let last_match_row = *last_row.get(&char_t).unwrap_or(&0);
-            let cost = if char_s == char_t { 0 } else { 1 };
-
-            let dist_add = matrix[(row, col + 1)] + 1;
-            let dist_del = matrix[(row + 1, col)] + 1;
-            let dist_sub = matrix[(row, col)] + cost;
-            let dist_trans = matrix[(last_match_row, last_match_col)]
-                + (row - last_match_row - 1)
-                + 1
-                + (col - last_match_col - 1);
-
-            let dist = min(min(dist_add, dist_del), min(dist_sub, dist_trans));
-
-            matrix[(row + 1, col + 1)] = dist;
-
-            if cost == 0 {
-                last_match_col = col;
-            }
-        }
-
-        last_row.insert(char_s, row);
-    }
-
-    let mut minimum = (u32::max_value(), 0);
-
-    for x in n..=m {
-        let dist = matrix[(n + 1, x + 1)] as u32;
-        if dist < minimum.0 {
-            minimum = (dist, x)
-        }
-    }
-
-    minimum
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn matched_length() {
-        let query = "Levenste";
-        let text = "Levenshtein";
-
-        let (dist, length) = prefix_damerau_levenshtein(query.as_bytes(), text.as_bytes());
-        assert_eq!(dist, 1);
-        assert_eq!(&text[..length], "Levenshte");
-    }
-
-    #[test]
-    #[should_panic]
-    fn matched_length_panic() {
-        let query = "Levenshtein";
-        let text = "Levenste";
-
-        // this function will panic if source if longer than target
-        prefix_damerau_levenshtein(query.as_bytes(), text.as_bytes());
-    }
-}

meilisearch-core/src/lib.rs

@@ -1,202 +0,0 @@
-#![allow(clippy::type_complexity)]
-
-#[cfg(test)]
-#[macro_use]
-extern crate assert_matches;
-#[macro_use]
-extern crate pest_derive;
-
-mod automaton;
-mod bucket_sort;
-mod database;
-mod distinct_map;
-mod error;
-mod filters;
-mod levenshtein;
-mod number;
-mod query_builder;
-mod query_tree;
-mod query_words_mapper;
-mod ranked_map;
-mod raw_document;
-mod reordered_attrs;
-pub mod criterion;
-pub mod facets;
-pub mod raw_indexer;
-pub mod serde;
-pub mod settings;
-pub mod store;
-pub mod update;
-
-pub use self::database::{BoxUpdateFn, Database, DatabaseOptions, MainT, UpdateT, MainWriter, MainReader, UpdateWriter, UpdateReader};
-pub use self::error::{Error, HeedError, FstError, MResult, pest_error, FacetError};
-pub use self::filters::Filter;
-pub use self::number::{Number, ParseNumberError};
-pub use self::ranked_map::RankedMap;
-pub use self::raw_document::RawDocument;
-pub use self::store::Index;
-pub use self::update::{EnqueuedUpdateResult, ProcessedUpdateResult, UpdateStatus, UpdateType};
-pub use meilisearch_types::{DocIndex, DocumentId, Highlight};
-pub use meilisearch_schema::Schema;
-pub use query_words_mapper::QueryWordsMapper;
-
-use compact_arena::SmallArena;
-use log::{error, trace};
-use std::borrow::Cow;
-use std::collections::HashMap;
-use std::convert::TryFrom;
-
-use crate::bucket_sort::PostingsListView;
-use crate::levenshtein::prefix_damerau_levenshtein;
-use crate::query_tree::{QueryId, QueryKind};
-use crate::reordered_attrs::ReorderedAttrs;
-
-type FstSetCow<'a> = fst::Set<Cow<'a, [u8]>>;
-type FstMapCow<'a> = fst::Map<Cow<'a, [u8]>>;
-
-#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
-pub struct Document {
-    pub id: DocumentId,
-    pub highlights: Vec<Highlight>,
-
-    #[cfg(test)]
-    pub matches: Vec<crate::bucket_sort::SimpleMatch>,
-}
-
-fn highlights_from_raw_document<'a, 'tag, 'txn>(
-    raw_document: &RawDocument<'a, 'tag>,
-    queries_kinds: &HashMap<QueryId, &QueryKind>,
-    arena: &SmallArena<'tag, PostingsListView<'txn>>,
-    searchable_attrs: Option<&ReorderedAttrs>,
-    schema: &Schema,
-) -> Vec<Highlight>
-{
-    let mut highlights = Vec::new();
-
-    for bm in raw_document.bare_matches.iter() {
-        let postings_list = &arena[bm.postings_list];
-        let input = postings_list.input();
-        let kind = &queries_kinds.get(&bm.query_index);
-
-        for di in postings_list.iter() {
-            let covered_area = match kind {
-                Some(QueryKind::NonTolerant(query)) | Some(QueryKind::Tolerant(query)) => {
-                    let len = if query.len() > input.len() {
-                        input.len()
-                    } else {
-                        prefix_damerau_levenshtein(query.as_bytes(), input).1
-                    };
-                    u16::try_from(len).unwrap_or(u16::max_value())
-                },
-                _ => di.char_length,
-            };
-
-            let attribute = searchable_attrs
-                .and_then(|sa| sa.reverse(di.attribute))
-                .unwrap_or(di.attribute);
-
-            let attribute = match schema.indexed_pos_to_field_id(attribute) {
-                Some(field_id) => field_id.0,
-                None => {
-                    error!("Cannot convert indexed_pos {} to field_id", attribute);
-                    trace!("Schema is compromized; {:?}", schema);
-                    continue
-                }
-            };
-
-            let highlight = Highlight {
-                attribute,
-                char_index: di.char_index,
-                char_length: covered_area,
-            };
-
-            highlights.push(highlight);
-        }
-    }
-
-    highlights
-}
-
-impl Document {
-    #[cfg(not(test))]
-    pub fn from_highlights(id: DocumentId, highlights: &[Highlight]) -> Document {
-        Document { id, highlights: highlights.to_owned() }
-    }
-
-    #[cfg(test)]
-    pub fn from_highlights(id: DocumentId, highlights: &[Highlight]) -> Document {
-        Document { id, highlights: highlights.to_owned(), matches: Vec::new() }
-    }
-
-    #[cfg(not(test))]
-    pub fn from_raw<'a, 'tag, 'txn>(
-        raw_document: RawDocument<'a, 'tag>,
-        queries_kinds: &HashMap<QueryId, &QueryKind>,
-        arena: &SmallArena<'tag, PostingsListView<'txn>>,
-        searchable_attrs: Option<&ReorderedAttrs>,
-        schema: &Schema,
-    ) -> Document
-    {
-        let highlights = highlights_from_raw_document(
-            &raw_document,
-            queries_kinds,
-            arena,
-            searchable_attrs,
-            schema,
-        );
-
-        Document { id: raw_document.id, highlights }
-    }
-
-    #[cfg(test)]
-    pub fn from_raw<'a, 'tag, 'txn>(
-        raw_document: RawDocument<'a, 'tag>,
-        queries_kinds: &HashMap<QueryId, &QueryKind>,
-        arena: &SmallArena<'tag, PostingsListView<'txn>>,
-        searchable_attrs: Option<&ReorderedAttrs>,
-        schema: &Schema,
-    ) -> Document
-    {
-        use crate::bucket_sort::SimpleMatch;
-
-        let highlights = highlights_from_raw_document(
-            &raw_document,
-            queries_kinds,
-            arena,
-            searchable_attrs,
-            schema,
-        );
-
-        let mut matches = Vec::new();
-        for sm in raw_document.processed_matches {
-            let attribute = searchable_attrs
-                .and_then(|sa| sa.reverse(sm.attribute))
-                .unwrap_or(sm.attribute);
-
-            let attribute = match schema.indexed_pos_to_field_id(attribute) {
-                Some(field_id) => field_id.0,
-                None => {
-                    error!("Cannot convert indexed_pos {} to field_id", attribute);
-                    trace!("Schema is compromized; {:?}", schema);
-                    continue
-                }
-            };
-
-            matches.push(SimpleMatch { attribute, ..sm });
-        }
-        matches.sort_unstable();
-
-        Document { id: raw_document.id, highlights, matches }
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use std::mem;
-
-    #[test]
-    fn docindex_mem_size() {
-        assert_eq!(mem::size_of::<DocIndex>(), 12);
-    }
-}

meilisearch-core/src/number.rs

@@ -1,120 +0,0 @@
-use std::cmp::Ordering;
-use std::fmt;
-use std::num::{ParseFloatError, ParseIntError};
-use std::str::FromStr;
-
-use ordered_float::OrderedFloat;
-use serde::{Deserialize, Serialize};
-
-#[derive(Serialize, Deserialize, Debug, Copy, Clone)]
-pub enum Number {
-    Unsigned(u64),
-    Signed(i64),
-    Float(OrderedFloat<f64>),
-    Null,
-}
-
-impl Default for Number {
-    fn default() -> Self {
-        Self::Null
-    }
-}
-
-impl FromStr for Number {
-    type Err = ParseNumberError;
-
-    fn from_str(s: &str) -> Result<Self, Self::Err> {
-        let uint_error = match u64::from_str(s) {
-            Ok(unsigned) => return Ok(Number::Unsigned(unsigned)),
-            Err(error) => error,
-        };
-
-        let int_error = match i64::from_str(s) {
-            Ok(signed) => return Ok(Number::Signed(signed)),
-            Err(error) => error,
-        };
-
-        let float_error = match f64::from_str(s) {
-            Ok(float) => return Ok(Number::Float(OrderedFloat(float))),
-            Err(error) => error,
-        };
-
-        Err(ParseNumberError {
-            uint_error,
-            int_error,
-            float_error,
-        })
-    }
-}
-
-impl PartialEq for Number {
-    fn eq(&self, other: &Number) -> bool {
-        self.cmp(other) == Ordering::Equal
-    }
-}
-
-impl Eq for Number {}
-
-impl PartialOrd for Number {
-    fn partial_cmp(&self, other: &Number) -> Option<Ordering> {
-        Some(self.cmp(other))
-    }
-}
-
-impl Ord for Number {
-    fn cmp(&self, other: &Self) -> Ordering {
-        use Number::{Float, Signed, Unsigned, Null};
-
-        match (*self, *other) {
-            (Unsigned(a), Unsigned(b)) => a.cmp(&b),
-            (Unsigned(a), Signed(b)) => {
-                if b < 0 {
-                    Ordering::Greater
-                } else {
-                    a.cmp(&(b as u64))
-                }
-            }
-            (Unsigned(a), Float(b)) => (OrderedFloat(a as f64)).cmp(&b),
-            (Signed(a), Unsigned(b)) => {
-                if a < 0 {
-                    Ordering::Less
-                } else {
-                    (a as u64).cmp(&b)
-                }
-            }
-            (Signed(a), Signed(b)) => a.cmp(&b),
-            (Signed(a), Float(b)) => OrderedFloat(a as f64).cmp(&b),
-            (Float(a), Unsigned(b)) => a.cmp(&OrderedFloat(b as f64)),
-            (Float(a), Signed(b)) => a.cmp(&OrderedFloat(b as f64)),
-            (Float(a), Float(b)) => a.cmp(&b),
-            (Null, Null) => Ordering::Equal,
-            (_, Null) => Ordering::Less,
-            (Null, _) => Ordering::Greater,
-        }
-    }
-}
-
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct ParseNumberError {
-    uint_error: ParseIntError,
-    int_error: ParseIntError,
-    float_error: ParseFloatError,
-}
-
-impl fmt::Display for ParseNumberError {
-    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        if self.uint_error == self.int_error {
-            write!(
-                f,
-                "can not parse number: {}, {}",
-                self.uint_error, self.float_error
-            )
-        } else {
-            write!(
-                f,
-                "can not parse number: {}, {}, {}",
-                self.uint_error, self.int_error, self.float_error
-            )
-        }
-    }
-}

meilisearch-core/src/query_tree.rs

@@ -1,560 +0,0 @@
-use std::borrow::Cow;
-use std::collections::HashMap;
-use std::hash::{Hash, Hasher};
-use std::ops::Range;
-use std::time::Instant;
-use std::{cmp, fmt, iter::once};
-
-use fst::{IntoStreamer, Streamer};
-use itertools::{EitherOrBoth, merge_join_by};
-use meilisearch_tokenizer::split_query_string;
-use sdset::{Set, SetBuf, SetOperation};
-use log::debug;
-
-use crate::database::MainT;
-use crate::{store, DocumentId, DocIndex, MResult, FstSetCow};
-use crate::automaton::{normalize_str, build_dfa, build_prefix_dfa, build_exact_dfa};
-use crate::QueryWordsMapper;
-
-#[derive(Clone, PartialEq, Eq, Hash)]
-pub enum Operation {
-    And(Vec<Operation>),
-    Or(Vec<Operation>),
-    Query(Query),
-}
-
-impl fmt::Debug for Operation {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        fn pprint_tree(f: &mut fmt::Formatter<'_>, op: &Operation, depth: usize) -> fmt::Result {
-            match op {
-                Operation::And(children) => {
-                    writeln!(f, "{:1$}AND", "", depth * 2)?;
-                    children.iter().try_for_each(|c| pprint_tree(f, c, depth + 1))
-                },
-                Operation::Or(children) => {
-                    writeln!(f, "{:1$}OR", "", depth * 2)?;
-                    children.iter().try_for_each(|c| pprint_tree(f, c, depth + 1))
-                },
-                Operation::Query(query) => writeln!(f, "{:2$}{:?}", "", query, depth * 2),
-            }
-        }
-
-        pprint_tree(f, self, 0)
-    }
-}
-
-impl Operation {
-    fn tolerant(id: QueryId, prefix: bool, s: &str) -> Operation {
-        Operation::Query(Query { id, prefix, exact: true, kind: QueryKind::Tolerant(s.to_string()) })
-    }
-
-    fn non_tolerant(id: QueryId, prefix: bool, s: &str) -> Operation {
-        Operation::Query(Query { id, prefix, exact: true, kind: QueryKind::NonTolerant(s.to_string()) })
-    }
-
-    fn phrase2(id: QueryId, prefix: bool, (left, right): (&str, &str)) -> Operation {
-        let kind = QueryKind::Phrase(vec![left.to_owned(), right.to_owned()]);
-        Operation::Query(Query { id, prefix, exact: true, kind })
-    }
-}
-
-pub type QueryId = usize;
-
-#[derive(Clone, Eq)]
-pub struct Query {
-    pub id: QueryId,
-    pub prefix: bool,
-    pub exact: bool,
-    pub kind: QueryKind,
-}
-
-impl PartialEq for Query {
-    fn eq(&self, other: &Self) -> bool {
-        self.prefix == other.prefix && self.kind == other.kind
-    }
-}
-
-impl Hash for Query {
-    fn hash<H: Hasher>(&self, state: &mut H) {
-        self.prefix.hash(state);
-        self.kind.hash(state);
-    }
-}
-
-#[derive(Clone, PartialEq, Eq, Hash)]
-pub enum QueryKind {
-    Tolerant(String),
-    NonTolerant(String),
-    Phrase(Vec<String>),
-}
-
-impl fmt::Debug for Query {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        let Query { id, prefix, kind, .. } = self;
-        let prefix = if *prefix { String::from("Prefix") } else { String::default() };
-        match kind {
-            QueryKind::NonTolerant(word) => {
-                f.debug_struct(&(prefix + "NonTolerant")).field("id", &id).field("word", &word).finish()
-            },
-            QueryKind::Tolerant(word) => {
-                f.debug_struct(&(prefix + "Tolerant")).field("id", &id).field("word", &word).finish()
-            },
-            QueryKind::Phrase(words) => {
-                f.debug_struct(&(prefix + "Phrase")).field("id", &id).field("words", &words).finish()
-            },
-        }
-    }
-}
-
-#[derive(Debug, Default)]
-pub struct PostingsList {
-    docids: SetBuf<DocumentId>,
-    matches: SetBuf<DocIndex>,
-}
-
-pub struct Context<'a> {
-    pub words_set: FstSetCow<'a>,
-    pub stop_words: FstSetCow<'a>,
-    pub synonyms: store::Synonyms,
-    pub postings_lists: store::PostingsLists,
-    pub prefix_postings_lists: store::PrefixPostingsListsCache,
-}
-
-fn split_best_frequency<'a>(reader: &heed::RoTxn<MainT>, ctx: &Context, word: &'a str) -> MResult<Option<(&'a str, &'a str)>> {
-    let chars = word.char_indices().skip(1);
-    let mut best = None;
-
-    for (i, _) in chars {
-        let (left, right) = word.split_at(i);
-
-        let left_freq = ctx.postings_lists
-            .postings_list(reader, left.as_bytes())?
-            .map(|p| p.docids.len())
-            .unwrap_or(0);
-        let right_freq = ctx.postings_lists
-            .postings_list(reader, right.as_bytes())?
-            .map(|p| p.docids.len())
-            .unwrap_or(0);
-
-        let min_freq = cmp::min(left_freq, right_freq);
-        if min_freq != 0 && best.map_or(true, |(old, _, _)| min_freq > old) {
-            best = Some((min_freq, left, right));
-        }
-    }
-
-    Ok(best.map(|(_, l, r)| (l, r)))
-}
-
-fn fetch_synonyms(reader: &heed::RoTxn<MainT>, ctx: &Context, words: &[&str]) -> MResult<Vec<Vec<String>>> {
-    let words = normalize_str(&words.join(" "));
-    let set = ctx.synonyms.synonyms_fst(reader, words.as_bytes())?;
-
-    let mut strings = Vec::new();
-    let mut stream = set.stream();
-    while let Some(input) = stream.next() {
-        if let Ok(input) = std::str::from_utf8(input) {
-            let alts = input.split_ascii_whitespace().map(ToOwned::to_owned).collect();
-            strings.push(alts);
-        }
-    }
-
-    Ok(strings)
-}
-
-fn create_operation<I, F>(iter: I, f: F) -> Operation
-where I: IntoIterator<Item=Operation>,
-      F: Fn(Vec<Operation>) -> Operation,
-{
-    let mut iter = iter.into_iter();
-    match (iter.next(), iter.next()) {
-        (Some(first), None) => first,
-        (first, second) => f(first.into_iter().chain(second).chain(iter).collect()),
-    }
-}
-
-const MAX_NGRAM: usize = 3;
-
-pub fn create_query_tree(
-    reader: &heed::RoTxn<MainT>,
-    ctx: &Context,
-    query: &str,
-) -> MResult<(Operation, HashMap<QueryId, Range<usize>>)>
-{
-    let words = split_query_string(query).map(str::to_lowercase);
-    let words = words.filter(|w| !ctx.stop_words.contains(w));
-    let words: Vec<_> = words.enumerate().collect();
-
-    let mut mapper = QueryWordsMapper::new(words.iter().map(|(_, w)| w));
-
-    fn create_inner(
-        reader: &heed::RoTxn<MainT>,
-        ctx: &Context,
-        mapper: &mut QueryWordsMapper,
-        words: &[(usize, String)],
-    ) -> MResult<Vec<Operation>>
-    {
-        let mut alts = Vec::new();
-
-        for ngram in 1..=MAX_NGRAM {
-            if let Some(group) = words.get(..ngram) {
-                let mut group_ops = Vec::new();
-
-                let tail = &words[ngram..];
-                let is_last = tail.is_empty();
-
-                let mut group_alts = Vec::new();
-                match group {
-                    [(id, word)] => {
-                        let mut idgen = ((id + 1) * 100)..;
-                        let range = (*id)..id+1;
-
-                        let phrase = split_best_frequency(reader, ctx, word)?
-                            .map(|ws| {
-                                let id = idgen.next().unwrap();
-                                idgen.next().unwrap();
-                                mapper.declare(range.clone(), id, &[ws.0, ws.1]);
-                                Operation::phrase2(id, is_last, ws)
-                            });
-
-                        let synonyms = fetch_synonyms(reader, ctx, &[word])?
-                            .into_iter()
-                            .map(|alts| {
-                                let exact = alts.len() == 1;
-                                let id = idgen.next().unwrap();
-                                mapper.declare(range.clone(), id, &alts);
-
-                                let mut idgen = once(id).chain(&mut idgen);
-                                let iter = alts.into_iter().map(|w| {
-                                    let id = idgen.next().unwrap();
-                                    let kind = QueryKind::NonTolerant(w);
-                                    Operation::Query(Query { id, prefix: false, exact, kind })
-                                });
-
-                                create_operation(iter, Operation::And)
-                            });
-
-                        let original = Operation::tolerant(*id, is_last, word);
-
-                        group_alts.push(original);
-                        group_alts.extend(synonyms.chain(phrase));
-                    },
-                    words => {
-                        let id = words[0].0;
-                        let mut idgen = ((id + 1) * 100_usize.pow(ngram as u32))..;
-                        let range = id..id+ngram;
-
-                        let words: Vec<_> = words.iter().map(|(_, s)| s.as_str()).collect();
-
-                        for synonym in fetch_synonyms(reader, ctx, &words)? {
-                            let exact = synonym.len() == 1;
-                            let id = idgen.next().unwrap();
-                            mapper.declare(range.clone(), id, &synonym);
-
-                            let mut idgen = once(id).chain(&mut idgen);
-                            let synonym = synonym.into_iter().map(|s| {
-                                let id = idgen.next().unwrap();
-                                let kind = QueryKind::NonTolerant(s);
-                                Operation::Query(Query { id, prefix: false, exact, kind })
-                            });
-                            group_alts.push(create_operation(synonym, Operation::And));
-                        }
-
-                        let id = idgen.next().unwrap();
-                        let concat = words.concat();
-                        mapper.declare(range.clone(), id, &[&concat]);
-                        group_alts.push(Operation::non_tolerant(id, is_last, &concat));
-                    }
-                }
-
-                group_ops.push(create_operation(group_alts, Operation::Or));
-
-                if !tail.is_empty() {
-                    let tail_ops = create_inner(reader, ctx, mapper, tail)?;
-                    group_ops.push(create_operation(tail_ops, Operation::Or));
-                }
-
-                alts.push(create_operation(group_ops, Operation::And));
-            }
-        }
-
-        Ok(alts)
-    }
-
-    let alternatives = create_inner(reader, ctx, &mut mapper, &words)?;
-    let operation = Operation::Or(alternatives);
-    let mapping = mapper.mapping();
-
-    Ok((operation, mapping))
-}
-
-#[derive(Debug, Clone, PartialEq, Eq, Hash)]
-pub struct PostingsKey<'o> {
-    pub query: &'o Query,
-    pub input: Vec<u8>,
-    pub distance: u8,
-    pub is_exact: bool,
-}
-
-pub type Postings<'o, 'txn> = HashMap<PostingsKey<'o>, Cow<'txn, Set<DocIndex>>>;
-pub type Cache<'o, 'txn> = HashMap<&'o Operation, Cow<'txn, Set<DocumentId>>>;
-
-pub struct QueryResult<'o, 'txn> {
-    pub docids: Cow<'txn, Set<DocumentId>>,
-    pub queries: Postings<'o, 'txn>,
-}
-
-pub fn traverse_query_tree<'o, 'txn>(
-    reader: &'txn heed::RoTxn<MainT>,
-    ctx: &Context,
-    tree: &'o Operation,
-) -> MResult<QueryResult<'o, 'txn>>
-{
-    fn execute_and<'o, 'txn>(
-        reader: &'txn heed::RoTxn<MainT>,
-        ctx: &Context,
-        cache: &mut Cache<'o, 'txn>,
-        postings: &mut Postings<'o, 'txn>,
-        depth: usize,
-        operations: &'o [Operation],
-    ) -> MResult<Cow<'txn, Set<DocumentId>>>
-    {
-        debug!("{:1$}AND", "", depth * 2);
-
-        let before = Instant::now();
-        let mut results = Vec::new();
-
-        for op in operations {
-            if cache.get(op).is_none() {
-                let docids = match op {
-                    Operation::And(ops) => execute_and(reader, ctx, cache, postings, depth + 1, &ops)?,
-                    Operation::Or(ops) => execute_or(reader, ctx, cache, postings, depth + 1, &ops)?,
-                    Operation::Query(query) => execute_query(reader, ctx, postings, depth + 1, &query)?,
-                };
-                cache.insert(op, docids);
-            }
-        }
-
-        for op in operations {
-            if let Some(docids) = cache.get(op) {
-                results.push(docids.as_ref());
-            }
-        }
-
-        let op = sdset::multi::Intersection::new(results);
-        let docids = op.into_set_buf();
-
-        debug!("{:3$}--- AND fetched {} documents in {:.02?}", "", docids.len(), before.elapsed(), depth * 2);
-
-        Ok(Cow::Owned(docids))
-    }
-
-    fn execute_or<'o, 'txn>(
-        reader: &'txn heed::RoTxn<MainT>,
-        ctx: &Context,
-        cache: &mut Cache<'o, 'txn>,
-        postings: &mut Postings<'o, 'txn>,
-        depth: usize,
-        operations: &'o [Operation],
-    ) -> MResult<Cow<'txn, Set<DocumentId>>>
-    {
-        debug!("{:1$}OR", "", depth * 2);
-
-        let before = Instant::now();
-        let mut results = Vec::new();
-
-        for op in operations {
-            if cache.get(op).is_none() {
-                let docids = match op {
-                    Operation::And(ops) => execute_and(reader, ctx, cache, postings, depth + 1, &ops)?,
-                    Operation::Or(ops) => execute_or(reader, ctx, cache, postings, depth + 1, &ops)?,
-                    Operation::Query(query) => execute_query(reader, ctx, postings, depth + 1, &query)?,
-                };
-                cache.insert(op, docids);
-            }
-        }
-
-        for op in operations {
-            if let Some(docids) = cache.get(op) {
-                results.push(docids.as_ref());
-            }
-        }
-
-        let op = sdset::multi::Union::new(results);
-        let docids = op.into_set_buf();
-
-        debug!("{:3$}--- OR fetched {} documents in {:.02?}", "", docids.len(), before.elapsed(), depth * 2);
-
-        Ok(Cow::Owned(docids))
-    }
-
-    fn execute_query<'o, 'txn>(
-        reader: &'txn heed::RoTxn<MainT>,
-        ctx: &Context,
-        postings: &mut Postings<'o, 'txn>,
-        depth: usize,
-        query: &'o Query,
-    ) -> MResult<Cow<'txn, Set<DocumentId>>>
-    {
-        let before = Instant::now();
-
-        let Query { prefix, kind, exact, .. } = query;
-        let docids: Cow<Set<_>> = match kind {
-            QueryKind::Tolerant(word) => {
-                if *prefix && word.len() <= 2 {
-                    let prefix = {
-                        let mut array = [0; 4];
-                        let bytes = word.as_bytes();
-                        array[..bytes.len()].copy_from_slice(bytes);
-                        array
-                    };
-
-                    // We retrieve the cached postings lists for all
-                    // the words that starts with this short prefix.
-                    let result = ctx.prefix_postings_lists.prefix_postings_list(reader, prefix)?.unwrap_or_default();
-                    let key = PostingsKey { query, input: word.clone().into_bytes(), distance: 0, is_exact: false };
-                    postings.insert(key, result.matches);
-                    let prefix_docids = &result.docids;
-
-                    // We retrieve the exact postings list for the prefix,
-                    // because we must consider these matches as exact.
-                    let result = ctx.postings_lists.postings_list(reader, word.as_bytes())?.unwrap_or_default();
-                    let key = PostingsKey { query, input: word.clone().into_bytes(), distance: 0, is_exact: true };
-                    postings.insert(key, result.matches);
-                    let exact_docids = &result.docids;
-
-                    let before = Instant::now();
-                    let docids = sdset::duo::Union::new(prefix_docids, exact_docids).into_set_buf();
-                    debug!("{:4$}prefix docids ({} and {}) construction took {:.02?}",
-                        "", prefix_docids.len(), exact_docids.len(), before.elapsed(), depth * 2);
-
-                    Cow::Owned(docids)
-
-                } else {
-                    let dfa = if *prefix { build_prefix_dfa(word) } else { build_dfa(word) };
-
-                    let byte = word.as_bytes()[0];
-                    let mut stream = if byte == u8::max_value() {
-                        ctx.words_set.search(&dfa).ge(&[byte]).into_stream()
-                    } else {
-                        ctx.words_set.search(&dfa).ge(&[byte]).lt(&[byte + 1]).into_stream()
-                    };
-
-                    let before = Instant::now();
-                    let mut results = Vec::new();
-                    while let Some(input) = stream.next() {
-                        if let Some(result) = ctx.postings_lists.postings_list(reader, input)? {
-                            let distance = dfa.eval(input).to_u8();
-                            let is_exact = *exact && distance == 0 && input.len() == word.len();
-                            results.push(result.docids);
-                            let key = PostingsKey { query, input: input.to_owned(), distance, is_exact };
-                            postings.insert(key, result.matches);
-                        }
-                    }
-                    debug!("{:3$}docids retrieval ({:?}) took {:.02?}", "", results.len(), before.elapsed(), depth * 2);
-
-                    let before = Instant::now();
-                    let docids = if results.len() > 10 {
-                        let cap = results.iter().map(|dis| dis.len()).sum();
-                        let mut docids = Vec::with_capacity(cap);
-                        for dis in results {
-                            docids.extend_from_slice(&dis);
-                        }
-                        SetBuf::from_dirty(docids)
-                    } else {
-                        let sets = results.iter().map(AsRef::as_ref).collect();
-                        sdset::multi::Union::new(sets).into_set_buf()
-                    };
-                    debug!("{:2$}docids construction took {:.02?}", "", before.elapsed(), depth * 2);
-
-                    Cow::Owned(docids)
-                }
-            },
-            QueryKind::NonTolerant(word) => {
-                // TODO support prefix and non-prefix exact DFA
-                let dfa = build_exact_dfa(word);
-
-                let byte = word.as_bytes()[0];
-                let mut stream = if byte == u8::max_value() {
-                    ctx.words_set.search(&dfa).ge(&[byte]).into_stream()
-                } else {
-                    ctx.words_set.search(&dfa).ge(&[byte]).lt(&[byte + 1]).into_stream()
-                };
-
-                let before = Instant::now();
-                let mut results = Vec::new();
-                while let Some(input) = stream.next() {
-                    if let Some(result) = ctx.postings_lists.postings_list(reader, input)? {
-                        let distance = dfa.eval(input).to_u8();
-                        results.push(result.docids);
-                        let key = PostingsKey { query, input: input.to_owned(), distance, is_exact: *exact };
-                        postings.insert(key, result.matches);
-                    }
-                }
-                debug!("{:3$}docids retrieval ({:?}) took {:.02?}", "", results.len(), before.elapsed(), depth * 2);
-
-                let before = Instant::now();
-                let docids = if results.len() > 10 {
-                    let cap = results.iter().map(|dis| dis.len()).sum();
-                    let mut docids = Vec::with_capacity(cap);
-                    for dis in results {
-                        docids.extend_from_slice(&dis);
-                    }
-                    SetBuf::from_dirty(docids)
-                } else {
-                    let sets = results.iter().map(AsRef::as_ref).collect();
-                    sdset::multi::Union::new(sets).into_set_buf()
-                };
-                debug!("{:2$}docids construction took {:.02?}", "", before.elapsed(), depth * 2);
-
-                Cow::Owned(docids)
-            },
-            QueryKind::Phrase(words) => {
-                // TODO support prefix and non-prefix exact DFA
-                if let [first, second] = words.as_slice() {
-                    let first = ctx.postings_lists.postings_list(reader, first.as_bytes())?.unwrap_or_default();
-                    let second = ctx.postings_lists.postings_list(reader, second.as_bytes())?.unwrap_or_default();
-
-                    let iter = merge_join_by(first.matches.as_slice(), second.matches.as_slice(), |a, b| {
-                        let x = (a.document_id, a.attribute, (a.word_index as u32) + 1);
-                        let y = (b.document_id, b.attribute, b.word_index as u32);
-                        x.cmp(&y)
-                    });
-
-                    let matches: Vec<_> = iter
-                        .filter_map(EitherOrBoth::both)
-                        .flat_map(|(a, b)| once(*a).chain(Some(*b)))
-                        .collect();
-
-                    let before = Instant::now();
-                    let mut docids: Vec<_> = matches.iter().map(|m| m.document_id).collect();
-                    docids.dedup();
-                    let docids = SetBuf::new(docids).unwrap();
-                    debug!("{:2$}docids construction took {:.02?}", "", before.elapsed(), depth * 2);
-
-                    let matches = Cow::Owned(SetBuf::from_dirty(matches));
-                    let key = PostingsKey { query, input: vec![], distance: 0, is_exact: true };
-                    postings.insert(key, matches);
-
-                    Cow::Owned(docids)
-                } else {
-                    debug!("{:2$}{:?} skipped", "", words, depth * 2);
-                    Cow::default()
-                }
-            },
-        };
-
-        debug!("{:4$}{:?} fetched {:?} documents in {:.02?}", "", query, docids.len(), before.elapsed(), depth * 2);
-        Ok(docids)
-    }
-
-    let mut cache = Cache::new();
-    let mut postings = Postings::new();
-
-    let docids = match tree {
-        Operation::And(ops) => execute_and(reader, ctx, &mut cache, &mut postings, 0, &ops)?,
-        Operation::Or(ops) => execute_or(reader, ctx, &mut cache, &mut postings, 0, &ops)?,
-        Operation::Query(query) => execute_query(reader, ctx, &mut postings, 0, &query)?,
-    };
-
-    Ok(QueryResult { docids, queries: postings })
-}

meilisearch-core/src/query_words_mapper.rs

@@ -1,416 +0,0 @@
-use std::collections::HashMap;
-use std::iter::FromIterator;
-use std::ops::Range;
-use intervaltree::{Element, IntervalTree};
-
-pub type QueryId = usize;
-
-pub struct QueryWordsMapper {
-    originals: Vec<String>,
-    mappings: HashMap<QueryId, (Range<usize>, Vec<String>)>,
-}
-
-impl QueryWordsMapper {
-    pub fn new<I, A>(originals: I) -> QueryWordsMapper
-    where I: IntoIterator<Item = A>,
-          A: ToString,
-    {
-        let originals = originals.into_iter().map(|s| s.to_string()).collect();
-        QueryWordsMapper { originals, mappings: HashMap::new() }
-    }
-
-    #[allow(clippy::len_zero)]
-    pub fn declare<I, A>(&mut self, range: Range<usize>, id: QueryId, replacement: I)
-    where I: IntoIterator<Item = A>,
-          A: ToString,
-    {
-        assert!(range.len() != 0);
-        assert!(self.originals.get(range.clone()).is_some());
-        assert!(id >= self.originals.len());
-
-        let replacement: Vec<_> = replacement.into_iter().map(|s| s.to_string()).collect();
-
-        assert!(!replacement.is_empty());
-
-        // We detect words at the end and at the front of the
-        // replacement that are common with the originals:
-        //
-        //     x a b c d e f g
-        //       ^^^/   \^^^
-        //     a b x c d k j e f
-        //     ^^^           ^^^
-        //
-
-        let left = &self.originals[..range.start];
-        let right = &self.originals[range.end..];
-
-        let common_left = longest_common_prefix(left, &replacement);
-        let common_right = longest_common_prefix(&replacement, right);
-
-        for i in 0..common_left {
-            let range = range.start - common_left + i..range.start - common_left + i + 1;
-            let replacement = vec![replacement[i].clone()];
-            self.mappings.insert(id + i, (range, replacement));
-        }
-
-        {
-            let replacement = replacement[common_left..replacement.len() - common_right].to_vec();
-            self.mappings.insert(id + common_left, (range.clone(), replacement));
-        }
-
-        for i in 0..common_right {
-            let id = id + replacement.len() - common_right + i;
-            let range = range.end + i..range.end + i + 1;
-            let replacement = vec![replacement[replacement.len() - common_right + i].clone()];
-            self.mappings.insert(id, (range, replacement));
-        }
-    }
-
-    pub fn mapping(self) -> HashMap<QueryId, Range<usize>> {
-        let mappings = self.mappings.into_iter().map(|(i, (r, v))| (r, (i, v)));
-        let intervals = IntervalTree::from_iter(mappings);
-
-        let mut output = HashMap::new();
-        let mut offset = 0;
-
-        // We map each original word to the biggest number of
-        // associated words.
-        for i in 0..self.originals.len() {
-            let max = intervals.query_point(i)
-                .filter_map(|e| {
-                    if e.range.end - 1 == i {
-                        let len = e.value.1.iter().skip(i - e.range.start).count();
-                        if len != 0 { Some(len) } else { None }
-                    } else { None }
-                })
-                .max()
-                .unwrap_or(1);
-
-            let range = i + offset..i + offset + max;
-            output.insert(i, range);
-            offset += max - 1;
-        }
-
-        // We retrieve the range that each original word
-        // is mapped to and apply it to each of the words.
-        for i in 0..self.originals.len() {
-
-            let iter = intervals.query_point(i).filter(|e| e.range.end - 1 == i);
-            for Element { range, value: (id, words) } in iter {
-
-                // We ask for the complete range mapped to the area we map.
-                let start = output.get(&range.start).map(|r| r.start).unwrap_or(range.start);
-                let end = output.get(&(range.end - 1)).map(|r| r.end).unwrap_or(range.end);
-                let range = start..end;
-
-                // We map each query id to one word until the last,
-                // we map it to the remainings words.
-                let add = range.len() - words.len();
-                for (j, x) in range.take(words.len()).enumerate() {
-                    let add = if j == words.len() - 1 { add } else { 0 }; // is last?
-                    let range = x..x + 1 + add;
-                    output.insert(id + j, range);
-                }
-            }
-        }
-
-        output
-    }
-}
-
-fn longest_common_prefix<T: Eq + std::fmt::Debug>(a: &[T], b: &[T]) -> usize {
-    let mut best = None;
-    for i in (0..a.len()).rev() {
-        let count = a[i..].iter().zip(b).take_while(|(a, b)| a == b).count();
-        best = match best {
-            Some(old) if count > old => Some(count),
-            Some(_) => break,
-            None => Some(count),
-        };
-    }
-    best.unwrap_or(0)
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn original_unmodified() {
-        let query = ["new", "york", "city", "subway"];
-        //             0       1       2        3
-        let mut builder = QueryWordsMapper::new(&query);
-
-        // new york = new york city
-        builder.declare(0..2, 4, &["new", "york", "city"]);
-        //                    ^      4       5       6
-
-        // new = new york city
-        builder.declare(0..1, 7, &["new", "york", "city"]);
-        //                    ^      7       8       9
-
-        let mapping = builder.mapping();
-
-        assert_eq!(mapping[&0], 0..1); // new
-        assert_eq!(mapping[&1], 1..2); // york
-        assert_eq!(mapping[&2], 2..3); // city
-        assert_eq!(mapping[&3], 3..4); // subway
-
-        assert_eq!(mapping[&4], 0..1); // new
-        assert_eq!(mapping[&5], 1..2); // york
-        assert_eq!(mapping[&6], 2..3); // city
-
-        assert_eq!(mapping[&7], 0..1); // new
-        assert_eq!(mapping[&8], 1..2); // york
-        assert_eq!(mapping[&9], 2..3); // city
-    }
-
-    #[test]
-    fn original_unmodified2() {
-        let query = ["new", "york", "city", "subway"];
-        //             0       1       2        3
-        let mut builder = QueryWordsMapper::new(&query);
-
-        // city subway = new york city underground train
-        builder.declare(2..4, 4, &["new", "york", "city", "underground", "train"]);
-        //                    ^      4      5       6           7           8
-
-        let mapping = builder.mapping();
-
-        assert_eq!(mapping[&0], 0..1); // new
-        assert_eq!(mapping[&1], 1..2); // york
-        assert_eq!(mapping[&2], 2..3); // city
-        assert_eq!(mapping[&3], 3..5); // subway
-
-        assert_eq!(mapping[&4], 0..1); // new
-        assert_eq!(mapping[&5], 1..2); // york
-        assert_eq!(mapping[&6], 2..3); // city
-        assert_eq!(mapping[&7], 3..4); // underground
-        assert_eq!(mapping[&8], 4..5); // train
-    }
-
-    #[test]
-    fn original_unmodified3() {
-        let query = ["a", "b", "x", "x", "a", "b", "c", "d", "e", "f", "g"];
-        //            0    1    2    3    4    5    6    7    8    9    10
-        let mut builder = QueryWordsMapper::new(&query);
-
-        // c d = a b x c d k j e f
-        builder.declare(6..8, 11, &["a", "b", "x", "c", "d", "k", "j", "e", "f"]);
-        //                    ^^    11   12   13   14   15   16   17   18   19
-
-        let mapping = builder.mapping();
-
-        assert_eq!(mapping[&0],  0..1); // a
-        assert_eq!(mapping[&1],  1..2); // b
-        assert_eq!(mapping[&2],  2..3); // x
-        assert_eq!(mapping[&3],  3..4); // x
-        assert_eq!(mapping[&4],  4..5); // a
-        assert_eq!(mapping[&5],  5..6); // b
-        assert_eq!(mapping[&6],  6..7); // c
-        assert_eq!(mapping[&7],  7..11); // d
-        assert_eq!(mapping[&8],  11..12); // e
-        assert_eq!(mapping[&9],  12..13); // f
-        assert_eq!(mapping[&10], 13..14); // g
-
-        assert_eq!(mapping[&11], 4..5); // a
-        assert_eq!(mapping[&12], 5..6); // b
-        assert_eq!(mapping[&13], 6..7); // x
-        assert_eq!(mapping[&14], 7..8); // c
-        assert_eq!(mapping[&15], 8..9); // d
-        assert_eq!(mapping[&16], 9..10); // k
-        assert_eq!(mapping[&17], 10..11); // j
-        assert_eq!(mapping[&18], 11..12); // e
-        assert_eq!(mapping[&19], 12..13); // f
-    }
-
-    #[test]
-    fn simple_growing() {
-        let query = ["new", "york", "subway"];
-        //             0       1        2
-        let mut builder = QueryWordsMapper::new(&query);
-
-        // new york = new york city
-        builder.declare(0..2, 3, &["new", "york", "city"]);
-        //                    ^      3       4       5
-
-        let mapping = builder.mapping();
-
-        assert_eq!(mapping[&0], 0..1); // new
-        assert_eq!(mapping[&1], 1..3); // york
-        assert_eq!(mapping[&2], 3..4); // subway
-        assert_eq!(mapping[&3], 0..1); // new
-        assert_eq!(mapping[&4], 1..2); // york
-        assert_eq!(mapping[&5], 2..3); // city
-    }
-
-    #[test]
-    fn same_place_growings() {
-        let query = ["NY", "subway"];
-        //             0       1
-        let mut builder = QueryWordsMapper::new(&query);
-
-        // NY = new york
-        builder.declare(0..1, 2, &["new", "york"]);
-        //                    ^      2       3
-
-        // NY = new york city
-        builder.declare(0..1, 4, &["new", "york", "city"]);
-        //                    ^      4       5       6
-
-        // NY = NYC
-        builder.declare(0..1, 7, &["NYC"]);
-        //                    ^      7
-
-        // NY = new york city
-        builder.declare(0..1, 8, &["new", "york", "city"]);
-        //                    ^      8       9      10
-
-        // subway = underground train
-        builder.declare(1..2, 11, &["underground", "train"]);
-        //                    ^          11          12
-
-        let mapping = builder.mapping();
-
-        assert_eq!(mapping[&0], 0..3); // NY
-        assert_eq!(mapping[&1], 3..5); // subway
-        assert_eq!(mapping[&2], 0..1); // new
-        assert_eq!(mapping[&3], 1..3); // york
-        assert_eq!(mapping[&4], 0..1); // new
-        assert_eq!(mapping[&5], 1..2); // york
-        assert_eq!(mapping[&6], 2..3); // city
-        assert_eq!(mapping[&7], 0..3); // NYC
-        assert_eq!(mapping[&8], 0..1); // new
-        assert_eq!(mapping[&9], 1..2); // york
-        assert_eq!(mapping[&10], 2..3); // city
-        assert_eq!(mapping[&11], 3..4); // underground
-        assert_eq!(mapping[&12], 4..5); // train
-    }
-
-    #[test]
-    fn bigger_growing() {
-        let query = ["NYC", "subway"];
-        //             0        1
-        let mut builder = QueryWordsMapper::new(&query);
-
-        // NYC = new york city
-        builder.declare(0..1, 2, &["new", "york", "city"]);
-        //                    ^      2       3       4
-
-        let mapping = builder.mapping();
-
-        assert_eq!(mapping[&0], 0..3); // NYC
-        assert_eq!(mapping[&1], 3..4); // subway
-        assert_eq!(mapping[&2], 0..1); // new
-        assert_eq!(mapping[&3], 1..2); // york
-        assert_eq!(mapping[&4], 2..3); // city
-    }
-
-    #[test]
-    fn middle_query_growing() {
-        let query = ["great", "awesome", "NYC", "subway"];
-        //              0         1        2        3
-        let mut builder = QueryWordsMapper::new(&query);
-
-        // NYC = new york city
-        builder.declare(2..3, 4, &["new", "york", "city"]);
-        //                    ^      4       5       6
-
-        let mapping = builder.mapping();
-
-        assert_eq!(mapping[&0], 0..1); // great
-        assert_eq!(mapping[&1], 1..2); // awesome
-        assert_eq!(mapping[&2], 2..5); // NYC
-        assert_eq!(mapping[&3], 5..6); // subway
-        assert_eq!(mapping[&4], 2..3); // new
-        assert_eq!(mapping[&5], 3..4); // york
-        assert_eq!(mapping[&6], 4..5); // city
-    }
-
-    #[test]
-    fn end_query_growing() {
-        let query = ["NYC", "subway"];
-        //             0        1
-        let mut builder = QueryWordsMapper::new(&query);
-
-        // NYC = new york city
-        builder.declare(1..2, 2, &["underground", "train"]);
-        //                    ^         2            3
-
-        let mapping = builder.mapping();
-
-        assert_eq!(mapping[&0], 0..1); // NYC
-        assert_eq!(mapping[&1], 1..3); // subway
-        assert_eq!(mapping[&2], 1..2); // underground
-        assert_eq!(mapping[&3], 2..3); // train
-    }
-
-    #[test]
-    fn multiple_growings() {
-        let query = ["great", "awesome", "NYC", "subway"];
-        //              0         1        2        3
-        let mut builder = QueryWordsMapper::new(&query);
-
-        // NYC = new york city
-        builder.declare(2..3, 4, &["new", "york", "city"]);
-        //                    ^      4       5       6
-
-        // subway = underground train
-        builder.declare(3..4, 7, &["underground", "train"]);
-        //                    ^          7           8
-
-        let mapping = builder.mapping();
-
-        assert_eq!(mapping[&0], 0..1); // great
-        assert_eq!(mapping[&1], 1..2); // awesome
-        assert_eq!(mapping[&2], 2..5); // NYC
-        assert_eq!(mapping[&3], 5..7); // subway
-        assert_eq!(mapping[&4], 2..3); // new
-        assert_eq!(mapping[&5], 3..4); // york
-        assert_eq!(mapping[&6], 4..5); // city
-        assert_eq!(mapping[&7], 5..6); // underground
-        assert_eq!(mapping[&8], 6..7); // train
-    }
-
-    #[test]
-    fn multiple_probable_growings() {
-        let query = ["great", "awesome", "NYC", "subway"];
-        //              0         1        2        3
-        let mut builder = QueryWordsMapper::new(&query);
-
-        // NYC = new york city
-        builder.declare(2..3, 4, &["new", "york", "city"]);
-        //                    ^      4       5       6
-
-        // subway = underground train
-        builder.declare(3..4, 7, &["underground", "train"]);
-        //                    ^          7           8
-
-        // great awesome = good
-        builder.declare(0..2, 9, &["good"]);
-        //                    ^       9
-
-        // awesome NYC = NY
-        builder.declare(1..3, 10, &["NY"]);
-        //                    ^^     10
-
-        // NYC subway = metro
-        builder.declare(2..4, 11, &["metro"]);
-        //                    ^^      11
-
-        let mapping = builder.mapping();
-
-        assert_eq!(mapping[&0], 0..1); // great
-        assert_eq!(mapping[&1], 1..2); // awesome
-        assert_eq!(mapping[&2], 2..5); // NYC
-        assert_eq!(mapping[&3], 5..7); // subway
-        assert_eq!(mapping[&4], 2..3); // new
-        assert_eq!(mapping[&5], 3..4); // york
-        assert_eq!(mapping[&6], 4..5); // city
-        assert_eq!(mapping[&7], 5..6); // underground
-        assert_eq!(mapping[&8], 6..7); // train
-        assert_eq!(mapping[&9], 0..2); // good
-        assert_eq!(mapping[&10], 1..5); // NY
-        assert_eq!(mapping[&11], 2..7); // metro
-    }
-}

meilisearch-core/src/ranked_map.rs

@@ -1,41 +0,0 @@
-use std::io::{Read, Write};
-
-use hashbrown::HashMap;
-use meilisearch_schema::FieldId;
-use serde::{Deserialize, Serialize};
-
-use crate::{DocumentId, Number};
-
-#[derive(Debug, Default, Clone, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(transparent)]
-pub struct RankedMap(HashMap<(DocumentId, FieldId), Number>);
-
-impl RankedMap {
-    pub fn len(&self) -> usize {
-        self.0.len()
-    }
-
-    pub fn is_empty(&self) -> bool {
-        self.0.is_empty()
-    }
-
-    pub fn insert(&mut self, document: DocumentId, field: FieldId, number: Number) {
-        self.0.insert((document, field), number);
-    }
-
-    pub fn remove(&mut self, document: DocumentId, field: FieldId) {
-        self.0.remove(&(document, field));
-    }
-
-    pub fn get(&self, document: DocumentId, field: FieldId) -> Option<Number> {
-        self.0.get(&(document, field)).cloned()
-    }
-
-    pub fn read_from_bin<R: Read>(reader: R) -> bincode::Result<RankedMap> {
-        bincode::deserialize_from(reader).map(RankedMap)
-    }
-
-    pub fn write_to_bin<W: Write>(&self, writer: W) -> bincode::Result<()> {
-        bincode::serialize_into(writer, &self.0)
-    }
-}

meilisearch-core/src/raw_document.rs

@@ -1,51 +0,0 @@
-use compact_arena::SmallArena;
-use sdset::SetBuf;
-use crate::DocIndex;
-use crate::bucket_sort::{SimpleMatch, BareMatch, PostingsListView};
-use crate::reordered_attrs::ReorderedAttrs;
-
-pub struct RawDocument<'a, 'tag> {
-    pub id: crate::DocumentId,
-    pub bare_matches: &'a mut [BareMatch<'tag>],
-    pub processed_matches: Vec<SimpleMatch>,
-    /// The list of minimum `distance` found
-    pub processed_distances: Vec<Option<u8>>,
-    /// Does this document contains a field
-    /// with one word that is exactly matching
-    pub contains_one_word_field: bool,
-}
-
-impl<'a, 'tag> RawDocument<'a, 'tag> {
-    pub fn new<'txn>(
-        bare_matches: &'a mut [BareMatch<'tag>],
-        postings_lists: &mut SmallArena<'tag, PostingsListView<'txn>>,
-        searchable_attrs: Option<&ReorderedAttrs>,
-    ) -> RawDocument<'a, 'tag>
-    {
-        if let Some(reordered_attrs) = searchable_attrs {
-            for bm in bare_matches.iter() {
-                let postings_list = &postings_lists[bm.postings_list];
-
-                let mut rewritten = Vec::new();
-                for di in postings_list.iter() {
-                    if let Some(attribute) = reordered_attrs.get(di.attribute) {
-                        rewritten.push(DocIndex { attribute, ..*di });
-                    }
-                }
-
-                let new_postings = SetBuf::from_dirty(rewritten);
-                postings_lists[bm.postings_list].rewrite_with(new_postings);
-            }
-        }
-
-        bare_matches.sort_unstable_by_key(|m| m.query_index);
-
-        RawDocument {
-            id: bare_matches[0].document_id,
-            bare_matches,
-            processed_matches: Vec::new(),
-            processed_distances: Vec::new(),
-            contains_one_word_field: false,
-        }
-    }
-}

meilisearch-core/src/raw_indexer.rs

@@ -1,312 +0,0 @@
-use std::borrow::Cow;
-use std::collections::{BTreeMap, HashMap};
-use std::convert::TryFrom;
-
-use deunicode::deunicode_with_tofu;
-use meilisearch_schema::IndexedPos;
-use meilisearch_tokenizer::{is_cjk, SeqTokenizer, Token, Tokenizer};
-use sdset::SetBuf;
-
-use crate::{DocIndex, DocumentId};
-use crate::FstSetCow;
-
-const WORD_LENGTH_LIMIT: usize = 80;
-
-type Word = Vec<u8>; // TODO make it be a SmallVec
-
-pub struct RawIndexer<A> {
-    word_limit: usize, // the maximum number of indexed words
-    stop_words: fst::Set<A>,
-    words_doc_indexes: BTreeMap<Word, Vec<DocIndex>>,
-    docs_words: HashMap<DocumentId, Vec<Word>>,
-}
-
-pub struct Indexed<'a> {
-    pub words_doc_indexes: BTreeMap<Word, SetBuf<DocIndex>>,
-    pub docs_words: HashMap<DocumentId, FstSetCow<'a>>,
-}
-
-impl<A> RawIndexer<A> {
-    pub fn new(stop_words: fst::Set<A>) -> RawIndexer<A> {
-        RawIndexer::with_word_limit(stop_words, 1000)
-    }
-
-    pub fn with_word_limit(stop_words: fst::Set<A>, limit: usize) -> RawIndexer<A> {
-        RawIndexer {
-            word_limit: limit,
-            stop_words,
-            words_doc_indexes: BTreeMap::new(),
-            docs_words: HashMap::new(),
-        }
-    }
-}
-
-impl<A: AsRef<[u8]>> RawIndexer<A> {
-    pub fn index_text(&mut self, id: DocumentId, indexed_pos: IndexedPos, text: &str) -> usize {
-        let mut number_of_words = 0;
-
-        for token in Tokenizer::new(text) {
-            let must_continue = index_token(
-                token,
-                id,
-                indexed_pos,
-                self.word_limit,
-                &self.stop_words,
-                &mut self.words_doc_indexes,
-                &mut self.docs_words,
-            );
-
-            number_of_words += 1;
-
-            if !must_continue {
-                break;
-            }
-        }
-
-        number_of_words
-    }
-
-    pub fn index_text_seq<'s, I>(&mut self, id: DocumentId, indexed_pos: IndexedPos, iter: I)
-    where
-        I: IntoIterator<Item = &'s str>,
-    {
-        let iter = iter.into_iter();
-        for token in SeqTokenizer::new(iter) {
-            let must_continue = index_token(
-                token,
-                id,
-                indexed_pos,
-                self.word_limit,
-                &self.stop_words,
-                &mut self.words_doc_indexes,
-                &mut self.docs_words,
-            );
-
-            if !must_continue {
-                break;
-            }
-        }
-    }
-
-    pub fn build(self) -> Indexed<'static> {
-        let words_doc_indexes = self
-            .words_doc_indexes
-            .into_iter()
-            .map(|(word, indexes)| (word, SetBuf::from_dirty(indexes)))
-            .collect();
-
-        let docs_words = self
-            .docs_words
-            .into_iter()
-            .map(|(id, mut words)| {
-                words.sort_unstable();
-                words.dedup();
-                let fst = fst::Set::from_iter(words).unwrap().map_data(Cow::Owned).unwrap();
-                (id, fst)
-            })
-            .collect();
-
-        Indexed {
-            words_doc_indexes,
-            docs_words,
-        }
-    }
-}
-
-fn index_token<A>(
-    token: Token,
-    id: DocumentId,
-    indexed_pos: IndexedPos,
-    word_limit: usize,
-    stop_words: &fst::Set<A>,
-    words_doc_indexes: &mut BTreeMap<Word, Vec<DocIndex>>,
-    docs_words: &mut HashMap<DocumentId, Vec<Word>>,
-) -> bool
-where A: AsRef<[u8]>,
-{
-    if token.index >= word_limit {
-        return false;
-    }
-
-    let lower = token.word.to_lowercase();
-    let token = Token {
-        word: &lower,
-        ..token
-    };
-
-    if !stop_words.contains(&token.word) {
-        match token_to_docindex(id, indexed_pos, token) {
-            Some(docindex) => {
-                let word = Vec::from(token.word);
-
-                if word.len() <= WORD_LENGTH_LIMIT {
-                    words_doc_indexes
-                        .entry(word.clone())
-                        .or_insert_with(Vec::new)
-                        .push(docindex);
-                    docs_words.entry(id).or_insert_with(Vec::new).push(word);
-
-                    if !lower.contains(is_cjk) {
-                        let unidecoded = deunicode_with_tofu(&lower, "");
-                        if unidecoded != lower && !unidecoded.is_empty() {
-                            let word = Vec::from(unidecoded);
-                            if word.len() <= WORD_LENGTH_LIMIT {
-                                words_doc_indexes
-                                    .entry(word.clone())
-                                    .or_insert_with(Vec::new)
-                                    .push(docindex);
-                                docs_words.entry(id).or_insert_with(Vec::new).push(word);
-                            }
-                        }
-                    }
-                }
-            }
-            None => return false,
-        }
-    }
-
-    true
-}
-
-fn token_to_docindex(id: DocumentId, indexed_pos: IndexedPos, token: Token) -> Option<DocIndex> {
-    let word_index = u16::try_from(token.word_index).ok()?;
-    let char_index = u16::try_from(token.char_index).ok()?;
-    let char_length = u16::try_from(token.word.chars().count()).ok()?;
-
-    let docindex = DocIndex {
-        document_id: id,
-        attribute: indexed_pos.0,
-        word_index,
-        char_index,
-        char_length,
-    };
-
-    Some(docindex)
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use meilisearch_schema::IndexedPos;
-
-    #[test]
-    fn strange_apostrophe() {
-        let mut indexer = RawIndexer::new(fst::Set::default());
-
-        let docid = DocumentId(0);
-        let indexed_pos = IndexedPos(0);
-        let text = "Zut, l’aspirateur, j’ai oublié de l’éteindre !";
-        indexer.index_text(docid, indexed_pos, text);
-
-        let Indexed {
-            words_doc_indexes, ..
-        } = indexer.build();
-
-        assert!(words_doc_indexes.get(&b"l"[..]).is_some());
-        assert!(words_doc_indexes.get(&b"aspirateur"[..]).is_some());
-        assert!(words_doc_indexes.get(&b"ai"[..]).is_some());
-        assert!(words_doc_indexes.get(&b"eteindre"[..]).is_some());
-        assert!(words_doc_indexes
-            .get(&"éteindre".to_owned().into_bytes())
-            .is_some());
-    }
-
-    #[test]
-    fn strange_apostrophe_in_sequence() {
-        let mut indexer = RawIndexer::new(fst::Set::default());
-
-        let docid = DocumentId(0);
-        let indexed_pos = IndexedPos(0);
-        let text = vec!["Zut, l’aspirateur, j’ai oublié de l’éteindre !"];
-        indexer.index_text_seq(docid, indexed_pos, text);
-
-        let Indexed {
-            words_doc_indexes, ..
-        } = indexer.build();
-
-        assert!(words_doc_indexes.get(&b"l"[..]).is_some());
-        assert!(words_doc_indexes.get(&b"aspirateur"[..]).is_some());
-        assert!(words_doc_indexes.get(&b"ai"[..]).is_some());
-        assert!(words_doc_indexes.get(&b"eteindre"[..]).is_some());
-        assert!(words_doc_indexes
-            .get(&"éteindre".to_owned().into_bytes())
-            .is_some());
-    }
-
-    #[test]
-    fn basic_stop_words() {
-        let stop_words = sdset::SetBuf::from_dirty(vec!["l", "j", "ai", "de"]);
-        let stop_words = fst::Set::from_iter(stop_words).unwrap();
-
-        let mut indexer = RawIndexer::new(stop_words);
-
-        let docid = DocumentId(0);
-        let indexed_pos = IndexedPos(0);
-        let text = "Zut, l’aspirateur, j’ai oublié de l’éteindre !";
-        indexer.index_text(docid, indexed_pos, text);
-
-        let Indexed {
-            words_doc_indexes, ..
-        } = indexer.build();
-
-        assert!(words_doc_indexes.get(&b"l"[..]).is_none());
-        assert!(words_doc_indexes.get(&b"aspirateur"[..]).is_some());
-        assert!(words_doc_indexes.get(&b"j"[..]).is_none());
-        assert!(words_doc_indexes.get(&b"ai"[..]).is_none());
-        assert!(words_doc_indexes.get(&b"de"[..]).is_none());
-        assert!(words_doc_indexes.get(&b"eteindre"[..]).is_some());
-        assert!(words_doc_indexes
-            .get(&"éteindre".to_owned().into_bytes())
-            .is_some());
-    }
-
-    #[test]
-    fn no_empty_unidecode() {
-        let mut indexer = RawIndexer::new(fst::Set::default());
-
-        let docid = DocumentId(0);
-        let indexed_pos = IndexedPos(0);
-        let text = "🇯🇵";
-        indexer.index_text(docid, indexed_pos, text);
-
-        let Indexed {
-            words_doc_indexes, ..
-        } = indexer.build();
-
-        assert!(words_doc_indexes
-            .get(&"🇯🇵".to_owned().into_bytes())
-            .is_some());
-    }
-
-    #[test]
-    // test sample from 807
-    fn very_long_text() {
-        let mut indexer = RawIndexer::new(fst::Set::default());
-        let indexed_pos = IndexedPos(0);
-        let docid = DocumentId(0);
-        let text = " The locations block is the most powerful, and potentially most involved, section of the .platform.app.yaml file. It allows you to control how the application container responds to incoming requests at a very fine-grained level. Common patterns also vary between language containers due to the way PHP-FPM handles incoming requests.\nEach entry of the locations block is an absolute URI path (with leading /) and its value includes the configuration directives for how the web server should handle matching requests. That is, if your domain is example.com then '/' means &ldquo;requests for example.com/&rdquo;, while '/admin' means &ldquo;requests for example.com/admin&rdquo;. If multiple blocks could match an incoming request then the most-specific will apply.\nweb:locations:&#39;/&#39;:# Rules for all requests that don&#39;t otherwise match....&#39;/sites/default/files&#39;:# Rules for any requests that begin with /sites/default/files....The simplest possible locations configuration is one that simply passes all requests on to your application unconditionally:\nweb:locations:&#39;/&#39;:passthru:trueThat is, all requests to /* should be forwarded to the process started by web.commands.start above. Note that for PHP containers the passthru key must specify what PHP file the request should be forwarded to, and must also specify a docroot under which the file lives. For example:\nweb:locations:&#39;/&#39;:root:&#39;web&#39;passthru:&#39;/app.php&#39;This block will serve requests to / from the web directory in the application, and if a file doesn&rsquo;t exist on disk then the request will be forwarded to the /app.php script.\nA full list of the possible subkeys for locations is below.\n  root: The folder from which to serve static assets for this location relative to the application root. The application root is the directory in which the .platform.app.yaml file is located. Typical values for this property include public or web. Setting it to '' is not recommended, and its behavior may vary depending on the type of application. Absolute paths are not supported.\n  passthru: Whether to forward disallowed and missing resources from this location to the application and can be true, false or an absolute URI path (with leading /). The default value is false. For non-PHP applications it will generally be just true or false. In a PHP application this will typically be the front controller such as /index.php or /app.php. This entry works similar to mod_rewrite under Apache. Note: If the value of passthru does not begin with the same value as the location key it is under, the passthru may evaluate to another entry. That may be useful when you want different cache settings for different paths, for instance, but want missing files in all of them to map back to the same front controller. See the example block below.\n  index: The files to consider when serving a request for a directory: an array of file names or null. (typically ['index.html']). Note that in order for this to work, access to the static files named must be allowed by the allow or rules keys for this location.\n  expires: How long to allow static assets from this location to be cached (this enables the Cache-Control and Expires headers) and can be a time or -1 for no caching (default). Times can be suffixed with &ldquo;ms&rdquo; (milliseconds), &ldquo;s&rdquo; (seconds), &ldquo;m&rdquo; (minutes), &ldquo;h&rdquo; (hours), &ldquo;d&rdquo; (days), &ldquo;w&rdquo; (weeks), &ldquo;M&rdquo; (months, 30d) or &ldquo;y&rdquo; (years, 365d).\n  scripts: Whether to allow loading scripts in that location (true or false). This directive is only meaningful on PHP.\n  allow: Whether to allow serving files which don&rsquo;t match a rule (true or false, default: true).\n  headers: Any additional headers to apply to static assets. This section is a mapping of header names to header values. Responses from the application aren&rsquo;t affected, to avoid overlap with the application&rsquo;s own ability to include custom headers in the response.\n  rules: Specific overrides for a specific location. The key is a PCRE (regular expression) that is matched against the full request path.\n  request_buffering: Most application servers do not support chunked requests (e.g. fpm, uwsgi), so Platform.sh enables request_buffering by default to handle them. That default configuration would look like this if it was present in .platform.app.yaml:\nweb:locations:&#39;/&#39;:passthru:truerequest_buffering:enabled:truemax_request_size:250mIf the application server can already efficiently handle chunked requests, the request_buffering subkey can be modified to disable it entirely (enabled: false). Additionally, applications that frequently deal with uploads greater than 250MB in size can update the max_request_size key to the application&rsquo;s needs. Note that modifications to request_buffering will need to be specified at each location where it is desired.\n ";
-        indexer.index_text(docid, indexed_pos, text);
-        let Indexed {
-            words_doc_indexes, ..
-        } = indexer.build();
-        assert!(words_doc_indexes.get(&"buffering".to_owned().into_bytes()).is_some());
-    }
-
-    #[test]
-    fn words_over_index_1000_not_indexed() {
-        let mut indexer = RawIndexer::new(fst::Set::default());
-        let indexed_pos = IndexedPos(0);
-        let docid = DocumentId(0);
-        let mut text = String::with_capacity(5000);
-        for _ in 0..1000 {
-            text.push_str("less ");
-        }
-        text.push_str("more");
-        indexer.index_text(docid, indexed_pos, &text);
-        let Indexed {
-            words_doc_indexes, ..
-        } = indexer.build();
-        assert!(words_doc_indexes.get(&"less".to_owned().into_bytes()).is_some());
-        assert!(words_doc_indexes.get(&"more".to_owned().into_bytes()).is_none());
-    }
-}

meilisearch-core/src/reordered_attrs.rs

@@ -1,31 +0,0 @@
-use std::cmp;
-
-#[derive(Default, Clone)]
-pub struct ReorderedAttrs {
-    reorders: Vec<Option<u16>>,
-    reverse: Vec<u16>,
-}
-
-impl ReorderedAttrs {
-    pub fn new() -> ReorderedAttrs {
-        ReorderedAttrs { reorders: Vec::new(), reverse: Vec::new() }
-    }
-
-    pub fn insert_attribute(&mut self, attribute: u16) {
-        let new_len = cmp::max(attribute as usize + 1, self.reorders.len());
-        self.reorders.resize(new_len, None);
-        self.reorders[attribute as usize] = Some(self.reverse.len() as u16);
-        self.reverse.push(attribute);
-    }
-
-    pub fn get(&self, attribute: u16) -> Option<u16> {
-        match self.reorders.get(attribute as usize)? {
-            Some(attribute) => Some(*attribute),
-            None => None,
-        }
-    }
-
-    pub fn reverse(&self, attribute: u16) -> Option<u16> {
-        self.reverse.get(attribute as usize).copied()
-    }
-}

meilisearch-core/src/serde/deserializer.rs

@@ -1,161 +0,0 @@
-use std::collections::HashSet;
-use std::io::Cursor;
-use std::{error::Error, fmt};
-
-use meilisearch_schema::{Schema, FieldId};
-use serde::{de, forward_to_deserialize_any};
-use serde_json::de::IoRead as SerdeJsonIoRead;
-use serde_json::Deserializer as SerdeJsonDeserializer;
-use serde_json::Error as SerdeJsonError;
-
-use crate::database::MainT;
-use crate::store::DocumentsFields;
-use crate::DocumentId;
-
-#[derive(Debug)]
-pub enum DeserializerError {
-    SerdeJson(SerdeJsonError),
-    Zlmdb(heed::Error),
-    Custom(String),
-}
-
-impl de::Error for DeserializerError {
-    fn custom<T: fmt::Display>(msg: T) -> Self {
-        DeserializerError::Custom(msg.to_string())
-    }
-}
-
-impl fmt::Display for DeserializerError {
-    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        match self {
-            DeserializerError::SerdeJson(e) => write!(f, "serde json related error: {}", e),
-            DeserializerError::Zlmdb(e) => write!(f, "heed related error: {}", e),
-            DeserializerError::Custom(s) => f.write_str(s),
-        }
-    }
-}
-
-impl Error for DeserializerError {}
-
-impl From<SerdeJsonError> for DeserializerError {
-    fn from(error: SerdeJsonError) -> DeserializerError {
-        DeserializerError::SerdeJson(error)
-    }
-}
-
-impl From<heed::Error> for DeserializerError {
-    fn from(error: heed::Error) -> DeserializerError {
-        DeserializerError::Zlmdb(error)
-    }
-}
-
-pub struct Deserializer<'a> {
-    pub document_id: DocumentId,
-    pub reader: &'a heed::RoTxn<MainT>,
-    pub documents_fields: DocumentsFields,
-    pub schema: &'a Schema,
-    pub fields: Option<&'a HashSet<FieldId>>,
-}
-
-impl<'de, 'a, 'b> de::Deserializer<'de> for &'b mut Deserializer<'a> {
-    type Error = DeserializerError;
-
-    fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, Self::Error>
-    where
-        V: de::Visitor<'de>,
-    {
-        self.deserialize_option(visitor)
-    }
-
-    fn deserialize_option<V>(self, visitor: V) -> Result<V::Value, Self::Error>
-    where
-        V: de::Visitor<'de>,
-    {
-        self.deserialize_map(visitor)
-    }
-
-    fn deserialize_map<V>(self, visitor: V) -> Result<V::Value, Self::Error>
-    where
-        V: de::Visitor<'de>,
-    {
-        let mut error = None;
-
-        let iter = self
-            .documents_fields
-            .document_fields(self.reader, self.document_id)?
-            .filter_map(|result| {
-                let (attr, value) = match result {
-                    Ok(value) => value,
-                    Err(e) => {
-                        error = Some(e);
-                        return None;
-                    }
-                };
-
-                let is_displayed = self.schema.is_displayed(attr);
-                if is_displayed && self.fields.map_or(true, |f| f.contains(&attr)) {
-                    if let Some(attribute_name) = self.schema.name(attr) {
-                        let cursor = Cursor::new(value.to_owned());
-                        let ioread = SerdeJsonIoRead::new(cursor);
-                        let value = Value(SerdeJsonDeserializer::new(ioread));
-
-                        Some((attribute_name, value))
-                    } else {
-                        None
-                    }
-                } else {
-                    None
-                }
-            });
-
-        let mut iter = iter.peekable();
-
-        let result = match iter.peek() {
-            Some(_) => {
-                let map_deserializer = de::value::MapDeserializer::new(iter);
-                visitor
-                    .visit_some(map_deserializer)
-                    .map_err(DeserializerError::from)
-            }
-            None => visitor.visit_none(),
-        };
-
-        match error.take() {
-            Some(error) => Err(error.into()),
-            None => result,
-        }
-    }
-
-    forward_to_deserialize_any! {
-        bool i8 i16 i32 i64 i128 u8 u16 u32 u64 u128 f32 f64 char str string
-        bytes byte_buf unit unit_struct newtype_struct seq tuple
-        tuple_struct struct enum identifier ignored_any
-    }
-}
-
-struct Value(SerdeJsonDeserializer<SerdeJsonIoRead<Cursor<Vec<u8>>>>);
-
-impl<'de> de::IntoDeserializer<'de, SerdeJsonError> for Value {
-    type Deserializer = Self;
-
-    fn into_deserializer(self) -> Self::Deserializer {
-        self
-    }
-}
-
-impl<'de> de::Deserializer<'de> for Value {
-    type Error = SerdeJsonError;
-
-    fn deserialize_any<V>(mut self, visitor: V) -> Result<V::Value, Self::Error>
-    where
-        V: de::Visitor<'de>,
-    {
-        self.0.deserialize_any(visitor)
-    }
-
-    forward_to_deserialize_any! {
-        bool i8 i16 i32 i64 i128 u8 u16 u32 u64 u128 f32 f64 char str string
-        bytes byte_buf option unit unit_struct newtype_struct seq tuple
-        tuple_struct map struct enum identifier ignored_any
-    }
-}

meilisearch-core/src/serde/mod.rs

@@ -1,92 +0,0 @@
-mod deserializer;
-
-pub use self::deserializer::{Deserializer, DeserializerError};
-
-use std::{error::Error, fmt};
-
-use serde::ser;
-use serde_json::Error as SerdeJsonError;
-use meilisearch_schema::Error as SchemaError;
-
-use crate::ParseNumberError;
-
-#[derive(Debug)]
-pub enum SerializerError {
-    DocumentIdNotFound,
-    InvalidDocumentIdFormat,
-    Zlmdb(heed::Error),
-    SerdeJson(SerdeJsonError),
-    ParseNumber(ParseNumberError),
-    Schema(SchemaError),
-    UnserializableType { type_name: &'static str },
-    UnindexableType { type_name: &'static str },
-    UnrankableType { type_name: &'static str },
-    Custom(String),
-}
-
-impl ser::Error for SerializerError {
-    fn custom<T: fmt::Display>(msg: T) -> Self {
-        SerializerError::Custom(msg.to_string())
-    }
-}
-
-impl fmt::Display for SerializerError {
-    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        match self {
-            SerializerError::DocumentIdNotFound => {
-                f.write_str("Primary key is missing.")
-            }
-            SerializerError::InvalidDocumentIdFormat => {
-                f.write_str("a document primary key can be of type integer or string only composed of alphanumeric characters, hyphens (-) and underscores (_).")
-            }
-            SerializerError::Zlmdb(e) => write!(f, "heed related error: {}", e),
-            SerializerError::SerdeJson(e) => write!(f, "serde json error: {}", e),
-            SerializerError::ParseNumber(e) => {
-                write!(f, "error while trying to parse a number: {}", e)
-            }
-            SerializerError::Schema(e) => write!(f, "impossible to update schema: {}", e),
-            SerializerError::UnserializableType { type_name } => {
-                write!(f, "{} is not a serializable type", type_name)
-            }
-            SerializerError::UnindexableType { type_name } => {
-                write!(f, "{} is not an indexable type", type_name)
-            }
-            SerializerError::UnrankableType { type_name } => {
-                write!(f, "{} types can not be used for ranking", type_name)
-            }
-            SerializerError::Custom(s) => f.write_str(s),
-        }
-    }
-}
-
-impl Error for SerializerError {}
-
-impl From<String> for SerializerError {
-    fn from(value: String) -> SerializerError {
-        SerializerError::Custom(value)
-    }
-}
-
-impl From<SerdeJsonError> for SerializerError {
-    fn from(error: SerdeJsonError) -> SerializerError {
-        SerializerError::SerdeJson(error)
-    }
-}
-
-impl From<heed::Error> for SerializerError {
-    fn from(error: heed::Error) -> SerializerError {
-        SerializerError::Zlmdb(error)
-    }
-}
-
-impl From<ParseNumberError> for SerializerError {
-    fn from(error: ParseNumberError) -> SerializerError {
-        SerializerError::ParseNumber(error)
-    }
-}
-
-impl From<SchemaError> for SerializerError {
-    fn from(error: SchemaError) -> SerializerError {
-        SerializerError::Schema(error)
-   }
-}

meilisearch-core/src/settings.rs

@@ -1,183 +0,0 @@
-use std::collections::{BTreeMap, BTreeSet, HashSet};
-use std::str::FromStr;
-use std::iter::IntoIterator;
-
-use serde::{Deserialize, Deserializer, Serialize};
-use once_cell::sync::Lazy;
-
-use self::RankingRule::*;
-
-pub const DEFAULT_RANKING_RULES: [RankingRule; 6] = [Typo, Words, Proximity, Attribute, WordsPosition, Exactness];
-
-static RANKING_RULE_REGEX: Lazy<regex::Regex> = Lazy::new(|| {
-    regex::Regex::new(r"(asc|desc)\(([a-zA-Z0-9-_]*)\)").unwrap()
-});
-
-#[derive(Default, Clone, Serialize, Deserialize)]
-#[serde(rename_all = "camelCase", deny_unknown_fields)]
-pub struct Settings {
-    #[serde(default, deserialize_with = "deserialize_some")]
-    pub ranking_rules: Option<Option<Vec<String>>>,
-    #[serde(default, deserialize_with = "deserialize_some")]
-    pub distinct_attribute: Option<Option<String>>,
-    #[serde(default, deserialize_with = "deserialize_some")]
-    pub searchable_attributes: Option<Option<Vec<String>>>,
-    #[serde(default, deserialize_with = "deserialize_some")]
-    pub displayed_attributes: Option<Option<HashSet<String>>>,
-    #[serde(default, deserialize_with = "deserialize_some")]
-    pub stop_words: Option<Option<BTreeSet<String>>>,
-    #[serde(default, deserialize_with = "deserialize_some")]
-    pub synonyms: Option<Option<BTreeMap<String, Vec<String>>>>,
-    #[serde(default, deserialize_with = "deserialize_some")]
-    pub attributes_for_faceting: Option<Option<Vec<String>>>,
-}
-
-// Any value that is present is considered Some value, including null.
-fn deserialize_some<'de, T, D>(deserializer: D) -> Result<Option<T>, D::Error>
-    where T: Deserialize<'de>,
-          D: Deserializer<'de>
-{
-    Deserialize::deserialize(deserializer).map(Some)
-}
-
-impl Settings {
-    pub fn to_update(&self) -> Result<SettingsUpdate, RankingRuleConversionError> {
-        let settings = self.clone();
-
-        let ranking_rules = match settings.ranking_rules {
-            Some(Some(rules)) => UpdateState::Update(RankingRule::try_from_iter(rules.iter())?),
-            Some(None) => UpdateState::Clear,
-            None => UpdateState::Nothing,
-        };
-
-        Ok(SettingsUpdate {
-            ranking_rules,
-            distinct_attribute: settings.distinct_attribute.into(),
-            primary_key: UpdateState::Nothing,
-            searchable_attributes: settings.searchable_attributes.into(),
-            displayed_attributes: settings.displayed_attributes.into(),
-            stop_words: settings.stop_words.into(),
-            synonyms: settings.synonyms.into(),
-            attributes_for_faceting: settings.attributes_for_faceting.into(),
-        })
-    }
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub enum UpdateState<T> {
-    Update(T),
-    Clear,
-    Nothing,
-}
-
-impl <T> From<Option<Option<T>>> for UpdateState<T> {
-    fn from(opt: Option<Option<T>>) -> UpdateState<T> {
-        match opt {
-            Some(Some(t)) => UpdateState::Update(t),
-            Some(None) => UpdateState::Clear,
-            None => UpdateState::Nothing,
-        }
-    }
-}
-
-#[derive(Debug, Clone)]
-pub struct RankingRuleConversionError;
-
-impl std::fmt::Display for RankingRuleConversionError {
-    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
-        write!(f, "impossible to convert into RankingRule")
-    }
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub enum RankingRule {
-    Typo,
-    Words,
-    Proximity,
-    Attribute,
-    WordsPosition,
-    Exactness,
-    Asc(String),
-    Desc(String),
-}
-
-impl std::fmt::Display for RankingRule {
-    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
-        match self {
-            RankingRule::Typo => f.write_str("typo"),
-            RankingRule::Words => f.write_str("words"),
-            RankingRule::Proximity => f.write_str("proximity"),
-            RankingRule::Attribute => f.write_str("attribute"),
-            RankingRule::WordsPosition => f.write_str("wordsPosition"),
-            RankingRule::Exactness => f.write_str("exactness"),
-            RankingRule::Asc(field) => write!(f, "asc({})", field),
-            RankingRule::Desc(field) => write!(f, "desc({})", field),
-        }
-    }
-}
-
-impl FromStr for RankingRule {
-    type Err = RankingRuleConversionError;
-
-    fn from_str(s: &str) -> Result<Self, Self::Err> {
-        let rule = match s {
-            "typo" => RankingRule::Typo,
-            "words" => RankingRule::Words,
-            "proximity" => RankingRule::Proximity,
-            "attribute" => RankingRule::Attribute,
-            "wordsPosition" => RankingRule::WordsPosition,
-            "exactness" => RankingRule::Exactness,
-            _ => {
-                let captures = RANKING_RULE_REGEX.captures(s).ok_or(RankingRuleConversionError)?;
-                match (captures.get(1).map(|m| m.as_str()), captures.get(2)) {
-                    (Some("asc"), Some(field)) => RankingRule::Asc(field.as_str().to_string()),
-                    (Some("desc"), Some(field)) => RankingRule::Desc(field.as_str().to_string()),
-                    _ => return Err(RankingRuleConversionError)
-                }
-            }
-        };
-        Ok(rule)
-    }
-}
-
-impl RankingRule {
-    pub fn field(&self) -> Option<&str> {
-        match self {
-            RankingRule::Asc(field) | RankingRule::Desc(field) => Some(field),
-            _ => None,
-        }
-    }
-
-    pub fn try_from_iter(rules: impl IntoIterator<Item = impl AsRef<str>>) -> Result<Vec<RankingRule>, RankingRuleConversionError> {
-        rules.into_iter()
-            .map(|s| RankingRule::from_str(s.as_ref()))
-            .collect()
-    }
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct SettingsUpdate {
-    pub ranking_rules: UpdateState<Vec<RankingRule>>,
-    pub distinct_attribute: UpdateState<String>,
-    pub primary_key: UpdateState<String>,
-    pub searchable_attributes: UpdateState<Vec<String>>,
-    pub displayed_attributes: UpdateState<HashSet<String>>,
-    pub stop_words: UpdateState<BTreeSet<String>>,
-    pub synonyms: UpdateState<BTreeMap<String, Vec<String>>>,
-    pub attributes_for_faceting: UpdateState<Vec<String>>,
-}
-
-impl Default for SettingsUpdate {
-    fn default() -> Self {
-        Self {
-            ranking_rules: UpdateState::Nothing,
-            distinct_attribute: UpdateState::Nothing,
-            primary_key: UpdateState::Nothing,
-            searchable_attributes: UpdateState::Nothing,
-            displayed_attributes: UpdateState::Nothing,
-            stop_words: UpdateState::Nothing,
-            synonyms: UpdateState::Nothing,
-            attributes_for_faceting: UpdateState::Nothing,
-        }
-    }
-}

meilisearch-core/src/store/cow_set.rs

@@ -1,32 +0,0 @@
-use std::borrow::Cow;
-
-use heed::{types::CowSlice, BytesEncode, BytesDecode};
-use sdset::{Set, SetBuf};
-use zerocopy::{AsBytes, FromBytes};
-
-pub struct CowSet<T>(std::marker::PhantomData<T>);
-
-impl<'a, T: 'a> BytesEncode<'a> for CowSet<T>
-where
-    T: AsBytes,
-{
-    type EItem = Set<T>;
-
-    fn bytes_encode(item: &'a Self::EItem) -> Option<Cow<[u8]>> {
-        CowSlice::bytes_encode(item.as_slice())
-    }
-}
-
-impl<'a, T: 'a> BytesDecode<'a> for CowSet<T>
-where
-    T: FromBytes + Copy,
-{
-    type DItem = Cow<'a, Set<T>>;
-
-    fn bytes_decode(bytes: &'a [u8]) -> Option<Self::DItem> {
-        match CowSlice::<T>::bytes_decode(bytes)? {
-            Cow::Owned(vec) => Some(Cow::Owned(SetBuf::new_unchecked(vec))),
-            Cow::Borrowed(slice) => Some(Cow::Borrowed(Set::new_unchecked(slice))),
-        }
-    }
-}

meilisearch-core/src/store/docs_words.rs

@@ -1,43 +0,0 @@
-use std::borrow::Cow;
-
-use heed::Result as ZResult;
-use heed::types::{ByteSlice, OwnedType};
-
-use crate::database::MainT;
-use crate::{DocumentId, FstSetCow};
-use super::BEU32;
-
-#[derive(Copy, Clone)]
-pub struct DocsWords {
-    pub(crate) docs_words: heed::Database<OwnedType<BEU32>, ByteSlice>,
-}
-
-impl DocsWords {
-    pub fn put_doc_words(
-        self,
-        writer: &mut heed::RwTxn<MainT>,
-        document_id: DocumentId,
-        words: &FstSetCow,
-    ) -> ZResult<()> {
-        let document_id = BEU32::new(document_id.0);
-        let bytes = words.as_fst().as_bytes();
-        self.docs_words.put(writer, &document_id, bytes)
-    }
-
-    pub fn del_doc_words(self, writer: &mut heed::RwTxn<MainT>, document_id: DocumentId) -> ZResult<bool> {
-        let document_id = BEU32::new(document_id.0);
-        self.docs_words.delete(writer, &document_id)
-    }
-
-    pub fn clear(self, writer: &mut heed::RwTxn<MainT>) -> ZResult<()> {
-        self.docs_words.clear(writer)
-    }
-
-    pub fn doc_words(self, reader: &heed::RoTxn<MainT>, document_id: DocumentId) -> ZResult<FstSetCow> {
-        let document_id = BEU32::new(document_id.0);
-        match self.docs_words.get(reader, &document_id)? {
-            Some(bytes) => Ok(fst::Set::new(bytes).unwrap().map_data(Cow::Borrowed).unwrap()),
-            None => Ok(fst::Set::default().map_data(Cow::Owned).unwrap()),
-        }
-    }
-}

meilisearch-core/src/store/documents_fields.rs

@@ -1,79 +0,0 @@
-use heed::types::{ByteSlice, OwnedType};
-use crate::database::MainT;
-use heed::Result as ZResult;
-use meilisearch_schema::FieldId;
-
-use super::DocumentFieldStoredKey;
-use crate::DocumentId;
-
-#[derive(Copy, Clone)]
-pub struct DocumentsFields {
-    pub(crate) documents_fields: heed::Database<OwnedType<DocumentFieldStoredKey>, ByteSlice>,
-}
-
-impl DocumentsFields {
-    pub fn put_document_field(
-        self,
-        writer: &mut heed::RwTxn<MainT>,
-        document_id: DocumentId,
-        field: FieldId,
-        value: &[u8],
-    ) -> ZResult<()> {
-        let key = DocumentFieldStoredKey::new(document_id, field);
-        self.documents_fields.put(writer, &key, value)
-    }
-
-    pub fn del_all_document_fields(
-        self,
-        writer: &mut heed::RwTxn<MainT>,
-        document_id: DocumentId,
-    ) -> ZResult<usize> {
-        let start = DocumentFieldStoredKey::new(document_id, FieldId::min());
-        let end = DocumentFieldStoredKey::new(document_id, FieldId::max());
-        self.documents_fields.delete_range(writer, &(start..=end))
-    }
-
-    pub fn clear(self, writer: &mut heed::RwTxn<MainT>) -> ZResult<()> {
-        self.documents_fields.clear(writer)
-    }
-
-    pub fn document_attribute<'txn>(
-        self,
-        reader: &'txn heed::RoTxn<MainT>,
-        document_id: DocumentId,
-        field: FieldId,
-    ) -> ZResult<Option<&'txn [u8]>> {
-        let key = DocumentFieldStoredKey::new(document_id, field);
-        self.documents_fields.get(reader, &key)
-    }
-
-    pub fn document_fields<'txn>(
-        self,
-        reader: &'txn heed::RoTxn<MainT>,
-        document_id: DocumentId,
-    ) -> ZResult<DocumentFieldsIter<'txn>> {
-        let start = DocumentFieldStoredKey::new(document_id, FieldId::min());
-        let end = DocumentFieldStoredKey::new(document_id, FieldId::max());
-        let iter = self.documents_fields.range(reader, &(start..=end))?;
-        Ok(DocumentFieldsIter { iter })
-    }
-}
-
-pub struct DocumentFieldsIter<'txn> {
-    iter: heed::RoRange<'txn, OwnedType<DocumentFieldStoredKey>, ByteSlice>,
-}
-
-impl<'txn> Iterator for DocumentFieldsIter<'txn> {
-    type Item = ZResult<(FieldId, &'txn [u8])>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        match self.iter.next() {
-            Some(Ok((key, bytes))) => {
-                let field_id = FieldId(key.field_id.get());
-                Some(Ok((field_id, bytes)))
-            }
-            Some(Err(e)) => Some(Err(e)),
-            None => None,
-        }
-    }
-}

meilisearch-core/src/store/documents_fields_counts.rs

@@ -1,143 +0,0 @@
-use super::DocumentFieldIndexedKey;
-use crate::database::MainT;
-use crate::DocumentId;
-use heed::types::OwnedType;
-use heed::Result as ZResult;
-use meilisearch_schema::IndexedPos;
-use crate::MResult;
-
-#[derive(Copy, Clone)]
-pub struct DocumentsFieldsCounts {
-    pub(crate) documents_fields_counts: heed::Database<OwnedType<DocumentFieldIndexedKey>, OwnedType<u16>>,
-}
-
-impl DocumentsFieldsCounts {
-    pub fn put_document_field_count(
-        self,
-        writer: &mut heed::RwTxn<MainT>,
-        document_id: DocumentId,
-        attribute: IndexedPos,
-        value: u16,
-    ) -> ZResult<()> {
-        let key = DocumentFieldIndexedKey::new(document_id, attribute);
-        self.documents_fields_counts.put(writer, &key, &value)
-    }
-
-    pub fn del_all_document_fields_counts(
-        self,
-        writer: &mut heed::RwTxn<MainT>,
-        document_id: DocumentId,
-    ) -> ZResult<usize> {
-        let start = DocumentFieldIndexedKey::new(document_id, IndexedPos::min());
-        let end = DocumentFieldIndexedKey::new(document_id, IndexedPos::max());
-        self.documents_fields_counts.delete_range(writer, &(start..=end))
-    }
-
-    pub fn clear(self, writer: &mut heed::RwTxn<MainT>) -> ZResult<()> {
-        self.documents_fields_counts.clear(writer)
-    }
-
-    pub fn document_field_count(
-        self,
-        reader: &heed::RoTxn<MainT>,
-        document_id: DocumentId,
-        attribute: IndexedPos,
-    ) -> ZResult<Option<u16>> {
-        let key = DocumentFieldIndexedKey::new(document_id, attribute);
-        match self.documents_fields_counts.get(reader, &key)? {
-            Some(count) => Ok(Some(count)),
-            None => Ok(None),
-        }
-    }
-
-    pub fn document_fields_counts<'txn>(
-        self,
-        reader: &'txn heed::RoTxn<MainT>,
-        document_id: DocumentId,
-    ) -> ZResult<DocumentFieldsCountsIter<'txn>> {
-        let start = DocumentFieldIndexedKey::new(document_id, IndexedPos::min());
-        let end = DocumentFieldIndexedKey::new(document_id, IndexedPos::max());
-        let iter = self.documents_fields_counts.range(reader, &(start..=end))?;
-        Ok(DocumentFieldsCountsIter { iter })
-    }
-
-    pub fn documents_ids<'txn>(self, reader: &'txn heed::RoTxn<MainT>) -> MResult<DocumentsIdsIter<'txn>> {
-        let iter = self.documents_fields_counts.iter(reader)?;
-        Ok(DocumentsIdsIter {
-            last_seen_id: None,
-            iter,
-        })
-    }
-
-    pub fn all_documents_fields_counts<'txn>(
-        self,
-        reader: &'txn heed::RoTxn<MainT>,
-    ) -> ZResult<AllDocumentsFieldsCountsIter<'txn>> {
-        let iter = self.documents_fields_counts.iter(reader)?;
-        Ok(AllDocumentsFieldsCountsIter { iter })
-    }
-}
-
-pub struct DocumentFieldsCountsIter<'txn> {
-    iter: heed::RoRange<'txn, OwnedType<DocumentFieldIndexedKey>, OwnedType<u16>>,
-}
-
-impl Iterator for DocumentFieldsCountsIter<'_> {
-    type Item = ZResult<(IndexedPos, u16)>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        match self.iter.next() {
-            Some(Ok((key, count))) => {
-                let indexed_pos = IndexedPos(key.indexed_pos.get());
-                Some(Ok((indexed_pos, count)))
-            }
-            Some(Err(e)) => Some(Err(e)),
-            None => None,
-        }
-    }
-}
-
-pub struct DocumentsIdsIter<'txn> {
-    last_seen_id: Option<DocumentId>,
-    iter: heed::RoIter<'txn, OwnedType<DocumentFieldIndexedKey>, OwnedType<u16>>,
-}
-
-impl Iterator for DocumentsIdsIter<'_> {
-    type Item = MResult<DocumentId>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        for result in &mut self.iter {
-            match result {
-                Ok((key, _)) => {
-                    let document_id = DocumentId(key.docid.get());
-                    if Some(document_id) != self.last_seen_id {
-                        self.last_seen_id = Some(document_id);
-                        return Some(Ok(document_id));
-                    }
-                }
-                Err(e) => return Some(Err(e.into())),
-            }
-        }
-        None
-    }
-}
-
-pub struct AllDocumentsFieldsCountsIter<'txn> {
-    iter: heed::RoIter<'txn, OwnedType<DocumentFieldIndexedKey>, OwnedType<u16>>,
-}
-
-impl Iterator for AllDocumentsFieldsCountsIter<'_> {
-    type Item = ZResult<(DocumentId, IndexedPos, u16)>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        match self.iter.next() {
-            Some(Ok((key, count))) => {
-                let docid = DocumentId(key.docid.get());
-                let indexed_pos = IndexedPos(key.indexed_pos.get());
-                Some(Ok((docid, indexed_pos, count)))
-            }
-            Some(Err(e)) => Some(Err(e)),
-            None => None,
-        }
-    }
-}

meilisearch-core/src/store/documents_ids.rs

@@ -1,75 +0,0 @@
-use std::borrow::Cow;
-
-use heed::{BytesDecode, BytesEncode};
-use sdset::Set;
-
-use crate::DocumentId;
-use super::cow_set::CowSet;
-
-pub struct DocumentsIds;
-
-impl BytesEncode<'_> for DocumentsIds {
-    type EItem = Set<DocumentId>;
-
-    fn bytes_encode(item: &Self::EItem) -> Option<Cow<[u8]>> {
-        CowSet::bytes_encode(item)
-    }
-}
-
-impl<'a> BytesDecode<'a> for DocumentsIds {
-    type DItem = Cow<'a, Set<DocumentId>>;
-
-    fn bytes_decode(bytes: &'a [u8]) -> Option<Self::DItem> {
-        CowSet::bytes_decode(bytes)
-    }
-}
-
-pub struct DiscoverIds<'a> {
-    ids_iter: std::slice::Iter<'a, DocumentId>,
-    left_id: Option<u32>,
-    right_id: Option<u32>,
-    available_range: std::ops::Range<u32>,
-}
-
-impl DiscoverIds<'_> {
-    pub fn new(ids: &Set<DocumentId>) -> DiscoverIds {
-        let mut ids_iter = ids.iter();
-        let right_id = ids_iter.next().map(|id| id.0);
-        let available_range = 0..right_id.unwrap_or(u32::max_value());
-        DiscoverIds { ids_iter, left_id: None, right_id, available_range }
-    }
-}
-
-impl Iterator for DiscoverIds<'_> {
-    type Item = DocumentId;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        loop {
-            match self.available_range.next() {
-                // The available range gives us a new id, we return it.
-                Some(id) => return Some(DocumentId(id)),
-                // The available range is exhausted, we need to find the next one.
-                None if self.available_range.end == u32::max_value() => return None,
-                None => loop {
-                    self.left_id = self.right_id.take();
-                    self.right_id = self.ids_iter.next().map(|id| id.0);
-                    match (self.left_id, self.right_id) {
-                        // We found a gap in the used ids, we can yield all ids
-                        // until the end of the gap
-                        (Some(l), Some(r)) => if l.saturating_add(1) != r {
-                            self.available_range = (l + 1)..r;
-                            break;
-                        },
-                        // The last used id has been reached, we can use all ids
-                        // until u32 MAX
-                        (Some(l), None) => {
-                            self.available_range = l.saturating_add(1)..u32::max_value();
-                            break;
-                        },
-                        _ => (),
-                    }
-                },
-            }
-        }
-    }
-}

meilisearch-core/src/store/facets.rs

@@ -1,97 +0,0 @@
-use std::borrow::Cow;
-use std::collections::HashMap;
-use std::mem;
-
-use heed::{RwTxn, RoTxn, RoRange, types::Str, BytesEncode, BytesDecode};
-use sdset::{SetBuf, Set, SetOperation};
-
-use meilisearch_types::DocumentId;
-use meilisearch_schema::FieldId;
-
-use crate::MResult;
-use crate::database::MainT;
-use crate::facets::FacetKey;
-use super::cow_set::CowSet;
-
-/// contains facet info
-#[derive(Clone, Copy)]
-pub struct Facets {
-    pub(crate) facets: heed::Database<FacetKey, FacetData>,
-}
-
-pub struct FacetData;
-
-impl<'a> BytesEncode<'a> for FacetData {
-    type EItem = (&'a str, &'a Set<DocumentId>);
-
-    fn bytes_encode(item: &'a Self::EItem) -> Option<Cow<'a, [u8]>> {
-        // get size of the first item
-        let first_size =  item.0.as_bytes().len();
-        let size = mem::size_of::<u64>()
-            + first_size
-            + item.1.len() * mem::size_of::<DocumentId>();
-        let mut buffer = Vec::with_capacity(size);
-        // encode the length of the first item
-        buffer.extend_from_slice(&first_size.to_be_bytes());
-        buffer.extend_from_slice(Str::bytes_encode(&item.0)?.as_ref());
-        let second_slice = CowSet::bytes_encode(&item.1)?;
-        buffer.extend_from_slice(second_slice.as_ref());
-        Some(Cow::Owned(buffer))
-    }
-}
-
-impl<'a> BytesDecode<'a> for FacetData {
-    type DItem = (&'a str, Cow<'a, Set<DocumentId>>);
-
-    fn bytes_decode(bytes: &'a [u8]) -> Option<Self::DItem> {
-        const LEN: usize = mem::size_of::<u64>();
-        let mut size_buf = [0; LEN];
-        size_buf.copy_from_slice(bytes.get(0..LEN)?);
-        // decode size of the first item from the bytes
-        let first_size = usize::from_be_bytes(size_buf);
-        // decode first and second items
-        let first_item = Str::bytes_decode(bytes.get(LEN..(LEN + first_size))?)?;
-        let second_item = CowSet::bytes_decode(bytes.get((LEN + first_size)..)?)?;
-        Some((first_item, second_item))
-    }
-}
-
-impl Facets {
-    // we use sdset::SetBuf to ensure the docids are sorted.
-    pub fn put_facet_document_ids(&self, writer: &mut RwTxn<MainT>, facet_key: FacetKey, doc_ids: &Set<DocumentId>, facet_value: &str) -> MResult<()> {
-        Ok(self.facets.put(writer, &facet_key, &(facet_value, doc_ids))?)
-    }
-
-    pub fn field_document_ids<'txn>(&self, reader: &'txn RoTxn<MainT>, field_id: FieldId) -> MResult<RoRange<'txn, FacetKey, FacetData>> {
-        Ok(self.facets.prefix_iter(reader, &FacetKey::new(field_id, String::new()))?)
-    }
-
-    pub fn facet_document_ids<'txn>(&self, reader: &'txn RoTxn<MainT>, facet_key: &FacetKey) -> MResult<Option<(&'txn str,Cow<'txn, Set<DocumentId>>)>> {
-        Ok(self.facets.get(reader, &facet_key)?)
-    }
-
-    /// updates the facets  store, revmoving the documents from the facets provided in the
-    /// `facet_map` argument
-    pub fn remove(&self, writer: &mut RwTxn<MainT>, facet_map: HashMap<FacetKey, (String, Vec<DocumentId>)>) -> MResult<()> {
-        for (key, (name, document_ids)) in facet_map {
-            if let Some((_, old)) = self.facets.get(writer, &key)? {
-                let to_remove = SetBuf::from_dirty(document_ids);
-                let new = sdset::duo::OpBuilder::new(old.as_ref(), to_remove.as_set()).difference().into_set_buf();
-                self.facets.put(writer, &key, &(&name, new.as_set()))?;
-            }
-        }
-        Ok(())
-    }
-
-    pub fn add(&self, writer: &mut RwTxn<MainT>, facet_map: HashMap<FacetKey, (String, Vec<DocumentId>)>) -> MResult<()> {
-        for (key, (facet_name, document_ids)) in facet_map {
-            let set = SetBuf::from_dirty(document_ids);
-            self.put_facet_document_ids(writer, key, set.as_set(), &facet_name)?;
-        }
-        Ok(())
-    }
-
-    pub fn clear(self, writer: &mut heed::RwTxn<MainT>) -> MResult<()> {
-        Ok(self.facets.clear(writer)?)
-    }
-}

meilisearch-core/src/store/main.rs

@@ -1,320 +0,0 @@
-use std::borrow::Cow;
-use std::collections::HashMap;
-
-use chrono::{DateTime, Utc};
-use heed::types::{ByteSlice, OwnedType, SerdeBincode, Str, CowSlice};
-use meilisearch_schema::{FieldId, Schema};
-use meilisearch_types::DocumentId;
-use sdset::Set;
-
-use crate::database::MainT;
-use crate::{RankedMap, MResult};
-use crate::settings::RankingRule;
-use crate::{FstSetCow, FstMapCow};
-use super::{CowSet, DocumentsIds};
-
-const ATTRIBUTES_FOR_FACETING_KEY: &str = "attributes-for-faceting";
-const CREATED_AT_KEY: &str = "created-at";
-const CUSTOMS_KEY: &str = "customs";
-const DISTINCT_ATTRIBUTE_KEY: &str = "distinct-attribute";
-const EXTERNAL_DOCIDS_KEY: &str = "external-docids";
-const FIELDS_DISTRIBUTION_KEY: &str = "fields-distribution";
-const INTERNAL_DOCIDS_KEY: &str = "internal-docids";
-const NAME_KEY: &str = "name";
-const NUMBER_OF_DOCUMENTS_KEY: &str = "number-of-documents";
-const RANKED_MAP_KEY: &str = "ranked-map";
-const RANKING_RULES_KEY: &str = "ranking-rules";
-const SCHEMA_KEY: &str = "schema";
-const SORTED_DOCUMENT_IDS_CACHE_KEY: &str = "sorted-document-ids-cache";
-const STOP_WORDS_KEY: &str = "stop-words";
-const SYNONYMS_KEY: &str = "synonyms";
-const UPDATED_AT_KEY: &str = "updated-at";
-const WORDS_KEY: &str = "words";
-
-pub type FreqsMap = HashMap<String, usize>;
-type SerdeFreqsMap = SerdeBincode<FreqsMap>;
-type SerdeDatetime = SerdeBincode<DateTime<Utc>>;
-
-#[derive(Copy, Clone)]
-pub struct Main {
-    pub(crate) main: heed::PolyDatabase,
-}
-
-impl Main {
-    pub fn clear(self, writer: &mut heed::RwTxn<MainT>) -> MResult<()> {
-        Ok(self.main.clear(writer)?)
-    }
-
-    pub fn put_name(self, writer: &mut heed::RwTxn<MainT>, name: &str) -> MResult<()> {
-        Ok(self.main.put::<_, Str, Str>(writer, NAME_KEY, name)?)
-    }
-
-    pub fn name(self, reader: &heed::RoTxn<MainT>) -> MResult<Option<String>> {
-        Ok(self
-            .main
-            .get::<_, Str, Str>(reader, NAME_KEY)?
-            .map(|name| name.to_owned()))
-    }
-
-    pub fn put_created_at(self, writer: &mut heed::RwTxn<MainT>) -> MResult<()> {
-        Ok(self.main.put::<_, Str, SerdeDatetime>(writer, CREATED_AT_KEY, &Utc::now())?)
-    }
-
-    pub fn created_at(self, reader: &heed::RoTxn<MainT>) -> MResult<Option<DateTime<Utc>>> {
-        Ok(self.main.get::<_, Str, SerdeDatetime>(reader, CREATED_AT_KEY)?)
-    }
-
-    pub fn put_updated_at(self, writer: &mut heed::RwTxn<MainT>) -> MResult<()> {
-        Ok(self.main.put::<_, Str, SerdeDatetime>(writer, UPDATED_AT_KEY, &Utc::now())?)
-    }
-
-    pub fn updated_at(self, reader: &heed::RoTxn<MainT>) -> MResult<Option<DateTime<Utc>>> {
-        Ok(self.main.get::<_, Str, SerdeDatetime>(reader, UPDATED_AT_KEY)?)
-    }
-
-    pub fn put_internal_docids(self, writer: &mut heed::RwTxn<MainT>, ids: &sdset::Set<DocumentId>) -> MResult<()> {
-        Ok(self.main.put::<_, Str, DocumentsIds>(writer, INTERNAL_DOCIDS_KEY, ids)?)
-    }
-
-    pub fn internal_docids<'txn>(self, reader: &'txn heed::RoTxn<MainT>) -> MResult<Cow<'txn, sdset::Set<DocumentId>>> {
-        match self.main.get::<_, Str, DocumentsIds>(reader, INTERNAL_DOCIDS_KEY)? {
-            Some(ids) => Ok(ids),
-            None => Ok(Cow::default()),
-        }
-    }
-
-    pub fn merge_internal_docids(self, writer: &mut heed::RwTxn<MainT>, new_ids: &sdset::Set<DocumentId>) -> MResult<()> {
-        use sdset::SetOperation;
-
-        // We do an union of the old and new internal ids.
-        let internal_docids = self.internal_docids(writer)?;
-        let internal_docids = sdset::duo::Union::new(&internal_docids, new_ids).into_set_buf();
-        Ok(self.put_internal_docids(writer, &internal_docids)?)
-    }
-
-    pub fn remove_internal_docids(self, writer: &mut heed::RwTxn<MainT>, ids: &sdset::Set<DocumentId>) -> MResult<()> {
-        use sdset::SetOperation;
-
-        // We do a difference of the old and new internal ids.
-        let internal_docids = self.internal_docids(writer)?;
-        let internal_docids = sdset::duo::Difference::new(&internal_docids, ids).into_set_buf();
-        Ok(self.put_internal_docids(writer, &internal_docids)?)
-    }
-
-    pub fn put_external_docids<A>(self, writer: &mut heed::RwTxn<MainT>, ids: &fst::Map<A>) -> MResult<()>
-    where A: AsRef<[u8]>,
-    {
-        Ok(self.main.put::<_, Str, ByteSlice>(writer, EXTERNAL_DOCIDS_KEY, ids.as_fst().as_bytes())?)
-    }
-
-    pub fn merge_external_docids<A>(self, writer: &mut heed::RwTxn<MainT>, new_docids: &fst::Map<A>) -> MResult<()>
-    where A: AsRef<[u8]>,
-    {
-        use fst::{Streamer, IntoStreamer};
-
-        // Do an union of the old and the new set of external docids.
-        let external_docids = self.external_docids(writer)?;
-        let mut op = external_docids.op().add(new_docids.into_stream()).r#union();
-        let mut build = fst::MapBuilder::memory();
-        while let Some((docid, values)) = op.next() {
-            build.insert(docid, values[0].value).unwrap();
-        }
-        drop(op);
-
-        let external_docids = build.into_map();
-        Ok(self.put_external_docids(writer, &external_docids)?)
-    }
-
-    pub fn remove_external_docids<A>(self, writer: &mut heed::RwTxn<MainT>, ids: &fst::Map<A>) -> MResult<()>
-    where A: AsRef<[u8]>,
-    {
-        use fst::{Streamer, IntoStreamer};
-
-        // Do an union of the old and the new set of external docids.
-        let external_docids = self.external_docids(writer)?;
-        let mut op = external_docids.op().add(ids.into_stream()).difference();
-        let mut build = fst::MapBuilder::memory();
-        while let Some((docid, values)) = op.next() {
-            build.insert(docid, values[0].value).unwrap();
-        }
-        drop(op);
-
-        let external_docids = build.into_map();
-        self.put_external_docids(writer, &external_docids)
-    }
-
-    pub fn external_docids(self, reader: &heed::RoTxn<MainT>) -> MResult<FstMapCow> {
-        match self.main.get::<_, Str, ByteSlice>(reader, EXTERNAL_DOCIDS_KEY)? {
-            Some(bytes) => Ok(fst::Map::new(bytes).unwrap().map_data(Cow::Borrowed).unwrap()),
-            None => Ok(fst::Map::default().map_data(Cow::Owned).unwrap()),
-        }
-    }
-
-    pub fn external_to_internal_docid(self, reader: &heed::RoTxn<MainT>, external_docid: &str) -> MResult<Option<DocumentId>> {
-        let external_ids = self.external_docids(reader)?;
-        Ok(external_ids.get(external_docid).map(|id| DocumentId(id as u32)))
-    }
-
-    pub fn words_fst(self, reader: &heed::RoTxn<MainT>) -> MResult<FstSetCow> {
-        match self.main.get::<_, Str, ByteSlice>(reader, WORDS_KEY)? {
-            Some(bytes) => Ok(fst::Set::new(bytes).unwrap().map_data(Cow::Borrowed).unwrap()),
-            None => Ok(fst::Set::default().map_data(Cow::Owned).unwrap()),
-        }
-    }
-
-    pub fn put_words_fst<A: AsRef<[u8]>>(self, writer: &mut heed::RwTxn<MainT>, fst: &fst::Set<A>) -> MResult<()> {
-        Ok(self.main.put::<_, Str, ByteSlice>(writer, WORDS_KEY, fst.as_fst().as_bytes())?)
-    }
-
-    pub fn put_sorted_document_ids_cache(self, writer: &mut heed::RwTxn<MainT>, documents_ids: &[DocumentId]) -> MResult<()> {
-        Ok(self.main.put::<_, Str, CowSlice<DocumentId>>(writer, SORTED_DOCUMENT_IDS_CACHE_KEY, documents_ids)?)
-    }
-
-    pub fn sorted_document_ids_cache(self, reader: &heed::RoTxn<MainT>) -> MResult<Option<Cow<[DocumentId]>>> {
-        Ok(self.main.get::<_, Str, CowSlice<DocumentId>>(reader, SORTED_DOCUMENT_IDS_CACHE_KEY)?)
-    }
-
-    pub fn put_schema(self, writer: &mut heed::RwTxn<MainT>, schema: &Schema) -> MResult<()> {
-        Ok(self.main.put::<_, Str, SerdeBincode<Schema>>(writer, SCHEMA_KEY, schema)?)
-    }
-
-    pub fn schema(self, reader: &heed::RoTxn<MainT>) -> MResult<Option<Schema>> {
-        Ok(self.main.get::<_, Str, SerdeBincode<Schema>>(reader, SCHEMA_KEY)?)
-    }
-
-    pub fn delete_schema(self, writer: &mut heed::RwTxn<MainT>) -> MResult<bool> {
-        Ok(self.main.delete::<_, Str>(writer, SCHEMA_KEY)?)
-    }
-
-    pub fn put_ranked_map(self, writer: &mut heed::RwTxn<MainT>, ranked_map: &RankedMap) -> MResult<()> {
-        Ok(self.main.put::<_, Str, SerdeBincode<RankedMap>>(writer, RANKED_MAP_KEY, &ranked_map)?)
-    }
-
-    pub fn ranked_map(self, reader: &heed::RoTxn<MainT>) -> MResult<Option<RankedMap>> {
-        Ok(self.main.get::<_, Str, SerdeBincode<RankedMap>>(reader, RANKED_MAP_KEY)?)
-    }
-
-    pub fn put_synonyms_fst<A: AsRef<[u8]>>(self, writer: &mut heed::RwTxn<MainT>, fst: &fst::Set<A>) -> MResult<()> {
-        let bytes = fst.as_fst().as_bytes();
-        Ok(self.main.put::<_, Str, ByteSlice>(writer, SYNONYMS_KEY, bytes)?)
-    }
-
-    pub(crate) fn synonyms_fst(self, reader: &heed::RoTxn<MainT>) -> MResult<FstSetCow> {
-        match self.main.get::<_, Str, ByteSlice>(reader, SYNONYMS_KEY)? {
-            Some(bytes) => Ok(fst::Set::new(bytes).unwrap().map_data(Cow::Borrowed).unwrap()),
-            None => Ok(fst::Set::default().map_data(Cow::Owned).unwrap()),
-        }
-    }
-
-    pub fn synonyms(self, reader: &heed::RoTxn<MainT>) -> MResult<Vec<String>> {
-        let synonyms = self
-            .synonyms_fst(&reader)?
-            .stream()
-            .into_strs()?;
-        Ok(synonyms)
-    }
-
-    pub fn put_stop_words_fst<A: AsRef<[u8]>>(self, writer: &mut heed::RwTxn<MainT>, fst: &fst::Set<A>) -> MResult<()> {
-        let bytes = fst.as_fst().as_bytes();
-        Ok(self.main.put::<_, Str, ByteSlice>(writer, STOP_WORDS_KEY, bytes)?)
-    }
-
-    pub(crate) fn stop_words_fst(self, reader: &heed::RoTxn<MainT>) -> MResult<FstSetCow> {
-        match self.main.get::<_, Str, ByteSlice>(reader, STOP_WORDS_KEY)? {
-            Some(bytes) => Ok(fst::Set::new(bytes).unwrap().map_data(Cow::Borrowed).unwrap()),
-            None => Ok(fst::Set::default().map_data(Cow::Owned).unwrap()),
-        }
-    }
-
-    pub fn stop_words(self, reader: &heed::RoTxn<MainT>) -> MResult<Vec<String>> {
-        let stop_word_list = self
-            .stop_words_fst(reader)?
-            .stream()
-            .into_strs()?;
-        Ok(stop_word_list)
-    }
-
-    pub fn put_number_of_documents<F>(self, writer: &mut heed::RwTxn<MainT>, f: F) -> MResult<u64>
-    where
-        F: Fn(u64) -> u64,
-    {
-        let new = self.number_of_documents(&*writer).map(f)?;
-        self.main
-            .put::<_, Str, OwnedType<u64>>(writer, NUMBER_OF_DOCUMENTS_KEY, &new)?;
-        Ok(new)
-    }
-
-    pub fn number_of_documents(self, reader: &heed::RoTxn<MainT>) -> MResult<u64> {
-        match self
-            .main
-            .get::<_, Str, OwnedType<u64>>(reader, NUMBER_OF_DOCUMENTS_KEY)? {
-            Some(value) => Ok(value),
-            None => Ok(0),
-        }
-    }
-
-    pub fn put_fields_distribution(
-        self,
-        writer: &mut heed::RwTxn<MainT>,
-        fields_frequency: &FreqsMap,
-    ) -> MResult<()> {
-        Ok(self.main.put::<_, Str, SerdeFreqsMap>(writer, FIELDS_DISTRIBUTION_KEY, fields_frequency)?)
-    }
-
-    pub fn fields_distribution(&self, reader: &heed::RoTxn<MainT>) -> MResult<Option<FreqsMap>> {
-        match self
-            .main
-            .get::<_, Str, SerdeFreqsMap>(reader, FIELDS_DISTRIBUTION_KEY)?
-        {
-            Some(freqs) => Ok(Some(freqs)),
-            None => Ok(None),
-        }
-    }
-
-    pub fn attributes_for_faceting<'txn>(&self, reader: &'txn heed::RoTxn<MainT>) -> MResult<Option<Cow<'txn, Set<FieldId>>>> {
-        Ok(self.main.get::<_, Str, CowSet<FieldId>>(reader, ATTRIBUTES_FOR_FACETING_KEY)?)
-    }
-
-    pub fn put_attributes_for_faceting(self, writer: &mut heed::RwTxn<MainT>, attributes: &Set<FieldId>) -> MResult<()> {
-        Ok(self.main.put::<_, Str, CowSet<FieldId>>(writer, ATTRIBUTES_FOR_FACETING_KEY, attributes)?)
-    }
-
-    pub fn delete_attributes_for_faceting(self, writer: &mut heed::RwTxn<MainT>) -> MResult<bool> {
-        Ok(self.main.delete::<_, Str>(writer, ATTRIBUTES_FOR_FACETING_KEY)?)
-    }
-
-    pub fn ranking_rules(&self, reader: &heed::RoTxn<MainT>) -> MResult<Option<Vec<RankingRule>>> {
-        Ok(self.main.get::<_, Str, SerdeBincode<Vec<RankingRule>>>(reader, RANKING_RULES_KEY)?)
-    }
-
-    pub fn put_ranking_rules(self, writer: &mut heed::RwTxn<MainT>, value: &[RankingRule]) -> MResult<()> {
-        Ok(self.main.put::<_, Str, SerdeBincode<Vec<RankingRule>>>(writer, RANKING_RULES_KEY, &value.to_vec())?)
-    }
-
-    pub fn delete_ranking_rules(self, writer: &mut heed::RwTxn<MainT>) -> MResult<bool> {
-        Ok(self.main.delete::<_, Str>(writer, RANKING_RULES_KEY)?)
-    }
-
-    pub fn distinct_attribute(&self, reader: &heed::RoTxn<MainT>) -> MResult<Option<FieldId>> {
-        match self.main.get::<_, Str, OwnedType<u16>>(reader, DISTINCT_ATTRIBUTE_KEY)? {
-            Some(value) => Ok(Some(FieldId(value.to_owned()))),
-            None => Ok(None),
-        }
-    }
-
-    pub fn put_distinct_attribute(self, writer: &mut heed::RwTxn<MainT>, value: FieldId) -> MResult<()> {
-        Ok(self.main.put::<_, Str, OwnedType<u16>>(writer, DISTINCT_ATTRIBUTE_KEY, &value.0)?)
-    }
-
-    pub fn delete_distinct_attribute(self, writer: &mut heed::RwTxn<MainT>) -> MResult<bool> {
-        Ok(self.main.delete::<_, Str>(writer, DISTINCT_ATTRIBUTE_KEY)?)
-    }
-
-    pub fn put_customs(self, writer: &mut heed::RwTxn<MainT>, customs: &[u8]) -> MResult<()> {
-        Ok(self.main.put::<_, Str, ByteSlice>(writer, CUSTOMS_KEY, customs)?)
-    }
-
-    pub fn customs<'txn>(self, reader: &'txn heed::RoTxn<MainT>) -> MResult<Option<&'txn [u8]>> {
-        Ok(self.main.get::<_, Str, ByteSlice>(reader, CUSTOMS_KEY)?)
-    }
-}

meilisearch-core/src/store/mod.rs

@@ -1,522 +0,0 @@
-mod cow_set;
-mod docs_words;
-mod documents_ids;
-mod documents_fields;
-mod documents_fields_counts;
-mod facets;
-mod main;
-mod postings_lists;
-mod prefix_documents_cache;
-mod prefix_postings_lists_cache;
-mod synonyms;
-mod updates;
-mod updates_results;
-
-pub use self::cow_set::CowSet;
-pub use self::docs_words::DocsWords;
-pub use self::documents_fields::{DocumentFieldsIter, DocumentsFields};
-pub use self::documents_fields_counts::{DocumentFieldsCountsIter, DocumentsFieldsCounts, DocumentsIdsIter};
-pub use self::documents_ids::{DocumentsIds, DiscoverIds};
-pub use self::facets::Facets;
-pub use self::main::Main;
-pub use self::postings_lists::PostingsLists;
-pub use self::prefix_documents_cache::PrefixDocumentsCache;
-pub use self::prefix_postings_lists_cache::PrefixPostingsListsCache;
-pub use self::synonyms::Synonyms;
-pub use self::updates::Updates;
-pub use self::updates_results::UpdatesResults;
-
-use std::borrow::Cow;
-use std::collections::HashSet;
-use std::convert::TryInto;
-use std::{mem, ptr};
-
-use heed::{BytesEncode, BytesDecode};
-use meilisearch_schema::{IndexedPos, FieldId};
-use sdset::{Set, SetBuf};
-use serde::de::{self, Deserialize};
-use zerocopy::{AsBytes, FromBytes};
-
-use crate::criterion::Criteria;
-use crate::database::{MainT, UpdateT};
-use crate::database::{UpdateEvent, UpdateEventsEmitter};
-use crate::serde::Deserializer;
-use crate::settings::SettingsUpdate;
-use crate::{query_builder::QueryBuilder, update, DocIndex, DocumentId, Error, MResult};
-
-type BEU32 = zerocopy::U32<byteorder::BigEndian>;
-type BEU64 = zerocopy::U64<byteorder::BigEndian>;
-pub type BEU16 = zerocopy::U16<byteorder::BigEndian>;
-
-#[derive(Debug, Copy, Clone, AsBytes, FromBytes)]
-#[repr(C)]
-pub struct DocumentFieldIndexedKey {
-    docid: BEU32,
-    indexed_pos: BEU16,
-}
-
-impl DocumentFieldIndexedKey {
-    fn new(docid: DocumentId, indexed_pos: IndexedPos) -> DocumentFieldIndexedKey {
-        DocumentFieldIndexedKey {
-            docid: BEU32::new(docid.0),
-            indexed_pos: BEU16::new(indexed_pos.0),
-        }
-    }
-}
-
-#[derive(Debug, Copy, Clone, AsBytes, FromBytes)]
-#[repr(C)]
-pub struct DocumentFieldStoredKey {
-    docid: BEU32,
-    field_id: BEU16,
-}
-
-impl DocumentFieldStoredKey {
-    fn new(docid: DocumentId, field_id: FieldId) -> DocumentFieldStoredKey {
-        DocumentFieldStoredKey {
-            docid: BEU32::new(docid.0),
-            field_id: BEU16::new(field_id.0),
-        }
-    }
-}
-
-#[derive(Default, Debug)]
-pub struct Postings<'a> {
-    pub docids: Cow<'a, Set<DocumentId>>,
-    pub matches: Cow<'a, Set<DocIndex>>,
-}
-
-pub struct PostingsCodec;
-
-impl<'a> BytesEncode<'a> for PostingsCodec {
-    type EItem = Postings<'a>;
-
-    fn bytes_encode(item: &'a Self::EItem) -> Option<Cow<'a, [u8]>> {
-        let u64_size = mem::size_of::<u64>();
-        let docids_size = item.docids.len() * mem::size_of::<DocumentId>();
-        let matches_size = item.matches.len() * mem::size_of::<DocIndex>();
-
-        let mut buffer = Vec::with_capacity(u64_size + docids_size + matches_size);
-
-        let docids_len = item.docids.len() as u64;
-        buffer.extend_from_slice(&docids_len.to_be_bytes());
-        buffer.extend_from_slice(item.docids.as_bytes());
-        buffer.extend_from_slice(item.matches.as_bytes());
-
-        Some(Cow::Owned(buffer))
-    }
-}
-
-fn aligned_to(bytes: &[u8], align: usize) -> bool {
-    (bytes as *const _ as *const () as usize) % align == 0
-}
-
-fn from_bytes_to_set<'a, T: 'a>(bytes: &'a [u8]) -> Option<Cow<'a, Set<T>>>
-where T: Clone + FromBytes
-{
-    match zerocopy::LayoutVerified::<_, [T]>::new_slice(bytes) {
-        Some(layout) => Some(Cow::Borrowed(Set::new_unchecked(layout.into_slice()))),
-        None => {
-            let len = bytes.len();
-            let elem_size = mem::size_of::<T>();
-
-            // ensure that it is the alignment that is wrong
-            // and the length is valid
-            if len % elem_size == 0 && !aligned_to(bytes, mem::align_of::<T>()) {
-                let elems = len / elem_size;
-                let mut vec = Vec::<T>::with_capacity(elems);
-
-                unsafe {
-                    let dst = vec.as_mut_ptr() as *mut u8;
-                    ptr::copy_nonoverlapping(bytes.as_ptr(), dst, len);
-                    vec.set_len(elems);
-                }
-
-                return Some(Cow::Owned(SetBuf::new_unchecked(vec)));
-            }
-
-            None
-        }
-    }
-}
-
-impl<'a> BytesDecode<'a> for PostingsCodec {
-    type DItem = Postings<'a>;
-
-    fn bytes_decode(bytes: &'a [u8]) -> Option<Self::DItem> {
-        let u64_size = mem::size_of::<u64>();
-        let docid_size = mem::size_of::<DocumentId>();
-
-        let (len_bytes, bytes) = bytes.split_at(u64_size);
-        let docids_len = len_bytes.try_into().ok().map(u64::from_be_bytes)? as usize;
-        let docids_size = docids_len * docid_size;
-
-        let docids_bytes = &bytes[..docids_size];
-        let matches_bytes = &bytes[docids_size..];
-
-        let docids = from_bytes_to_set(docids_bytes)?;
-        let matches = from_bytes_to_set(matches_bytes)?;
-
-        Some(Postings { docids, matches })
-    }
-}
-
-fn main_name(name: &str) -> String {
-    format!("store-{}", name)
-}
-
-fn postings_lists_name(name: &str) -> String {
-    format!("store-{}-postings-lists", name)
-}
-
-fn documents_fields_name(name: &str) -> String {
-    format!("store-{}-documents-fields", name)
-}
-
-fn documents_fields_counts_name(name: &str) -> String {
-    format!("store-{}-documents-fields-counts", name)
-}
-
-fn synonyms_name(name: &str) -> String {
-    format!("store-{}-synonyms", name)
-}
-
-fn docs_words_name(name: &str) -> String {
-    format!("store-{}-docs-words", name)
-}
-
-fn prefix_documents_cache_name(name: &str) -> String {
-    format!("store-{}-prefix-documents-cache", name)
-}
-
-fn prefix_postings_lists_cache_name(name: &str) -> String {
-    format!("store-{}-prefix-postings-lists-cache", name)
-}
-
-fn updates_name(name: &str) -> String {
-    format!("store-{}-updates", name)
-}
-
-fn updates_results_name(name: &str) -> String {
-    format!("store-{}-updates-results", name)
-}
-
-fn facets_name(name: &str) -> String {
-    format!("store-{}-facets", name)
-}
-
-#[derive(Clone)]
-pub struct Index {
-    pub main: Main,
-    pub postings_lists: PostingsLists,
-    pub documents_fields: DocumentsFields,
-    pub documents_fields_counts: DocumentsFieldsCounts,
-    pub facets: Facets,
-    pub synonyms: Synonyms,
-    pub docs_words: DocsWords,
-    pub prefix_documents_cache: PrefixDocumentsCache,
-    pub prefix_postings_lists_cache: PrefixPostingsListsCache,
-
-    pub updates: Updates,
-    pub updates_results: UpdatesResults,
-    pub(crate) updates_notifier: UpdateEventsEmitter,
-}
-
-impl Index {
-    pub fn document<T: de::DeserializeOwned>(
-        &self,
-        reader: &heed::RoTxn<MainT>,
-        attributes: Option<&HashSet<&str>>,
-        document_id: DocumentId,
-    ) -> MResult<Option<T>> {
-        let schema = self.main.schema(reader)?;
-        let schema = schema.ok_or(Error::SchemaMissing)?;
-
-        let attributes = match attributes {
-            Some(attributes) => Some(attributes.iter().filter_map(|name| schema.id(*name)).collect()),
-            None => None,
-        };
-
-        let mut deserializer = Deserializer {
-            document_id,
-            reader,
-            documents_fields: self.documents_fields,
-            schema: &schema,
-            fields: attributes.as_ref(),
-        };
-
-        Ok(Option::<T>::deserialize(&mut deserializer)?)
-    }
-
-    pub fn document_attribute<T: de::DeserializeOwned>(
-        &self,
-        reader: &heed::RoTxn<MainT>,
-        document_id: DocumentId,
-        attribute: FieldId,
-    ) -> MResult<Option<T>> {
-        let bytes = self
-            .documents_fields
-            .document_attribute(reader, document_id, attribute)?;
-        match bytes {
-            Some(bytes) => Ok(Some(serde_json::from_slice(bytes)?)),
-            None => Ok(None),
-        }
-    }
-
-    pub fn document_attribute_bytes<'txn>(
-        &self,
-        reader: &'txn heed::RoTxn<MainT>,
-        document_id: DocumentId,
-        attribute: FieldId,
-    ) -> MResult<Option<&'txn [u8]>> {
-        let bytes = self
-            .documents_fields
-            .document_attribute(reader, document_id, attribute)?;
-        match bytes {
-            Some(bytes) => Ok(Some(bytes)),
-            None => Ok(None),
-        }
-    }
-
-    pub fn customs_update(&self, writer: &mut heed::RwTxn<UpdateT>, customs: Vec<u8>) -> MResult<u64> {
-        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
-        Ok(update::push_customs_update(writer, self.updates, self.updates_results, customs)?)
-    }
-
-    pub fn settings_update(&self, writer: &mut heed::RwTxn<UpdateT>, update: SettingsUpdate) -> MResult<u64> {
-        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
-        Ok(update::push_settings_update(writer, self.updates, self.updates_results, update)?)
-    }
-
-    pub fn documents_addition<D>(&self) -> update::DocumentsAddition<D> {
-        update::DocumentsAddition::new(
-            self.updates,
-            self.updates_results,
-            self.updates_notifier.clone(),
-        )
-    }
-
-    pub fn documents_partial_addition<D>(&self) -> update::DocumentsAddition<D> {
-        update::DocumentsAddition::new_partial(
-            self.updates,
-            self.updates_results,
-            self.updates_notifier.clone(),
-        )
-    }
-
-    pub fn documents_deletion(&self) -> update::DocumentsDeletion {
-        update::DocumentsDeletion::new(
-            self.updates,
-            self.updates_results,
-            self.updates_notifier.clone(),
-        )
-    }
-
-    pub fn clear_all(&self, writer: &mut heed::RwTxn<UpdateT>) -> MResult<u64> {
-        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
-        update::push_clear_all(writer, self.updates, self.updates_results)
-    }
-
-    pub fn current_update_id(&self, reader: &heed::RoTxn<UpdateT>) -> MResult<Option<u64>> {
-        match self.updates.last_update(reader)? {
-            Some((id, _)) => Ok(Some(id)),
-            None => Ok(None),
-        }
-    }
-
-    pub fn update_status(
-        &self,
-        reader: &heed::RoTxn<UpdateT>,
-        update_id: u64,
-    ) -> MResult<Option<update::UpdateStatus>> {
-        update::update_status(reader, self.updates, self.updates_results, update_id)
-    }
-
-    pub fn all_updates_status(&self, reader: &heed::RoTxn<UpdateT>) -> MResult<Vec<update::UpdateStatus>> {
-        let mut updates = Vec::new();
-        let mut last_update_result_id = 0;
-
-        // retrieve all updates results
-        if let Some((last_id, _)) = self.updates_results.last_update(reader)? {
-            updates.reserve(last_id as usize);
-
-            for id in 0..=last_id {
-                if let Some(update) = self.update_status(reader, id)? {
-                    updates.push(update);
-                    last_update_result_id = id + 1;
-                }
-            }
-        }
-
-        // retrieve all enqueued updates
-        if let Some((last_id, _)) = self.updates.last_update(reader)? {
-            for id in last_update_result_id..=last_id {
-                if let Some(update) = self.update_status(reader, id)? {
-                    updates.push(update);
-                }
-            }
-        }
-
-        Ok(updates)
-    }
-
-    pub fn query_builder(&self) -> QueryBuilder {
-        QueryBuilder::new(self)
-    }
-
-    pub fn query_builder_with_criteria<'c, 'f, 'd, 'i>(
-        &'i self,
-        criteria: Criteria<'c>,
-    ) -> QueryBuilder<'c, 'f, 'd, 'i> {
-        QueryBuilder::with_criteria(self, criteria)
-    }
-}
-
-pub fn create(
-    env: &heed::Env,
-    update_env: &heed::Env,
-    name: &str,
-    updates_notifier: UpdateEventsEmitter,
-) -> MResult<Index> {
-    // create all the store names
-    let main_name = main_name(name);
-    let postings_lists_name = postings_lists_name(name);
-    let documents_fields_name = documents_fields_name(name);
-    let documents_fields_counts_name = documents_fields_counts_name(name);
-    let synonyms_name = synonyms_name(name);
-    let docs_words_name = docs_words_name(name);
-    let prefix_documents_cache_name = prefix_documents_cache_name(name);
-    let prefix_postings_lists_cache_name = prefix_postings_lists_cache_name(name);
-    let updates_name = updates_name(name);
-    let updates_results_name = updates_results_name(name);
-    let facets_name = facets_name(name);
-
-    // open all the stores
-    let main = env.create_poly_database(Some(&main_name))?;
-    let postings_lists = env.create_database(Some(&postings_lists_name))?;
-    let documents_fields = env.create_database(Some(&documents_fields_name))?;
-    let documents_fields_counts = env.create_database(Some(&documents_fields_counts_name))?;
-    let facets = env.create_database(Some(&facets_name))?;
-    let synonyms = env.create_database(Some(&synonyms_name))?;
-    let docs_words = env.create_database(Some(&docs_words_name))?;
-    let prefix_documents_cache = env.create_database(Some(&prefix_documents_cache_name))?;
-    let prefix_postings_lists_cache = env.create_database(Some(&prefix_postings_lists_cache_name))?;
-    let updates = update_env.create_database(Some(&updates_name))?;
-    let updates_results = update_env.create_database(Some(&updates_results_name))?;
-
-    Ok(Index {
-        main: Main { main },
-        postings_lists: PostingsLists { postings_lists },
-        documents_fields: DocumentsFields { documents_fields },
-        documents_fields_counts: DocumentsFieldsCounts { documents_fields_counts },
-        synonyms: Synonyms { synonyms },
-        docs_words: DocsWords { docs_words },
-        prefix_postings_lists_cache: PrefixPostingsListsCache { prefix_postings_lists_cache },
-        prefix_documents_cache: PrefixDocumentsCache { prefix_documents_cache },
-        facets: Facets { facets },
-
-        updates: Updates { updates },
-        updates_results: UpdatesResults { updates_results },
-        updates_notifier,
-    })
-}
-
-pub fn open(
-    env: &heed::Env,
-    update_env: &heed::Env,
-    name: &str,
-    updates_notifier: UpdateEventsEmitter,
-) -> MResult<Option<Index>> {
-    // create all the store names
-    let main_name = main_name(name);
-    let postings_lists_name = postings_lists_name(name);
-    let documents_fields_name = documents_fields_name(name);
-    let documents_fields_counts_name = documents_fields_counts_name(name);
-    let synonyms_name = synonyms_name(name);
-    let docs_words_name = docs_words_name(name);
-    let prefix_documents_cache_name = prefix_documents_cache_name(name);
-    let facets_name = facets_name(name);
-    let prefix_postings_lists_cache_name = prefix_postings_lists_cache_name(name);
-    let updates_name = updates_name(name);
-    let updates_results_name = updates_results_name(name);
-
-    // open all the stores
-    let main = match env.open_poly_database(Some(&main_name))? {
-        Some(main) => main,
-        None => return Ok(None),
-    };
-    let postings_lists = match env.open_database(Some(&postings_lists_name))? {
-        Some(postings_lists) => postings_lists,
-        None => return Ok(None),
-    };
-    let documents_fields = match env.open_database(Some(&documents_fields_name))? {
-        Some(documents_fields) => documents_fields,
-        None => return Ok(None),
-    };
-    let documents_fields_counts = match env.open_database(Some(&documents_fields_counts_name))? {
-        Some(documents_fields_counts) => documents_fields_counts,
-        None => return Ok(None),
-    };
-    let synonyms = match env.open_database(Some(&synonyms_name))? {
-        Some(synonyms) => synonyms,
-        None => return Ok(None),
-    };
-    let docs_words = match env.open_database(Some(&docs_words_name))? {
-        Some(docs_words) => docs_words,
-        None => return Ok(None),
-    };
-    let prefix_documents_cache = match env.open_database(Some(&prefix_documents_cache_name))? {
-        Some(prefix_documents_cache) => prefix_documents_cache,
-        None => return Ok(None),
-    };
-    let facets = match env.open_database(Some(&facets_name))? {
-        Some(facets) => facets,
-        None => return Ok(None),
-    };
-    let prefix_postings_lists_cache = match env.open_database(Some(&prefix_postings_lists_cache_name))? {
-        Some(prefix_postings_lists_cache) => prefix_postings_lists_cache,
-        None => return Ok(None),
-    };
-    let updates = match update_env.open_database(Some(&updates_name))? {
-        Some(updates) => updates,
-        None => return Ok(None),
-    };
-    let updates_results = match update_env.open_database(Some(&updates_results_name))? {
-        Some(updates_results) => updates_results,
-        None => return Ok(None),
-    };
-
-    Ok(Some(Index {
-        main: Main { main },
-        postings_lists: PostingsLists { postings_lists },
-        documents_fields: DocumentsFields { documents_fields },
-        documents_fields_counts: DocumentsFieldsCounts { documents_fields_counts },
-        synonyms: Synonyms { synonyms },
-        docs_words: DocsWords { docs_words },
-        prefix_documents_cache: PrefixDocumentsCache { prefix_documents_cache },
-        facets: Facets { facets },
-        prefix_postings_lists_cache: PrefixPostingsListsCache { prefix_postings_lists_cache },
-        updates: Updates { updates },
-        updates_results: UpdatesResults { updates_results },
-        updates_notifier,
-    }))
-}
-
-pub fn clear(
-    writer: &mut heed::RwTxn<MainT>,
-    update_writer: &mut heed::RwTxn<UpdateT>,
-    index: &Index,
-) -> MResult<()> {
-    // clear all the stores
-    index.main.clear(writer)?;
-    index.postings_lists.clear(writer)?;
-    index.documents_fields.clear(writer)?;
-    index.documents_fields_counts.clear(writer)?;
-    index.synonyms.clear(writer)?;
-    index.docs_words.clear(writer)?;
-    index.prefix_documents_cache.clear(writer)?;
-    index.prefix_postings_lists_cache.clear(writer)?;
-    index.updates.clear(update_writer)?;
-    index.updates_results.clear(update_writer)?;
-    Ok(())
-}

meilisearch-core/src/store/postings_lists.rs

@@ -1,47 +0,0 @@
-use std::borrow::Cow;
-
-use heed::Result as ZResult;
-use heed::types::ByteSlice;
-use sdset::{Set, SetBuf};
-use slice_group_by::GroupBy;
-
-use crate::database::MainT;
-use crate::DocIndex;
-use crate::store::{Postings, PostingsCodec};
-
-#[derive(Copy, Clone)]
-pub struct PostingsLists {
-    pub(crate) postings_lists: heed::Database<ByteSlice, PostingsCodec>,
-}
-
-impl PostingsLists {
-    pub fn put_postings_list(
-        self,
-        writer: &mut heed::RwTxn<MainT>,
-        word: &[u8],
-        matches: &Set<DocIndex>,
-    ) -> ZResult<()> {
-        let docids = matches.linear_group_by_key(|m| m.document_id).map(|g| g[0].document_id).collect();
-        let docids = Cow::Owned(SetBuf::new_unchecked(docids));
-        let matches = Cow::Borrowed(matches);
-        let postings = Postings { docids, matches };
-
-        self.postings_lists.put(writer, word, &postings)
-    }
-
-    pub fn del_postings_list(self, writer: &mut heed::RwTxn<MainT>, word: &[u8]) -> ZResult<bool> {
-        self.postings_lists.delete(writer, word)
-    }
-
-    pub fn clear(self, writer: &mut heed::RwTxn<MainT>) -> ZResult<()> {
-        self.postings_lists.clear(writer)
-    }
-
-    pub fn postings_list<'txn>(
-        self,
-        reader: &'txn heed::RoTxn<MainT>,
-        word: &[u8],
-    ) -> ZResult<Option<Postings<'txn>>> {
-        self.postings_lists.get(reader, word)
-    }
-}

meilisearch-core/src/store/prefix_documents_cache.rs

@@ -1,80 +0,0 @@
-use std::borrow::Cow;
-
-use heed::types::{OwnedType, CowSlice};
-use heed::Result as ZResult;
-use zerocopy::{AsBytes, FromBytes};
-
-use super::{BEU64, BEU32};
-use crate::{DocumentId, Highlight};
-use crate::database::MainT;
-
-#[derive(Debug, Copy, Clone, AsBytes, FromBytes)]
-#[repr(C)]
-pub struct PrefixKey {
-    prefix: [u8; 4],
-    index: BEU64,
-    docid: BEU32,
-}
-
-impl PrefixKey {
-    pub fn new(prefix: [u8; 4], index: u64, docid: u32) -> PrefixKey {
-        PrefixKey {
-            prefix,
-            index: BEU64::new(index),
-            docid: BEU32::new(docid),
-        }
-    }
-}
-
-#[derive(Copy, Clone)]
-pub struct PrefixDocumentsCache {
-    pub(crate) prefix_documents_cache: heed::Database<OwnedType<PrefixKey>, CowSlice<Highlight>>,
-}
-
-impl PrefixDocumentsCache {
-    pub fn put_prefix_document(
-        self,
-        writer: &mut heed::RwTxn<MainT>,
-        prefix: [u8; 4],
-        index: usize,
-        docid: DocumentId,
-        highlights: &[Highlight],
-    ) -> ZResult<()> {
-        let key = PrefixKey::new(prefix, index as u64, docid.0);
-        self.prefix_documents_cache.put(writer, &key, highlights)
-    }
-
-    pub fn clear(self, writer: &mut heed::RwTxn<MainT>) -> ZResult<()> {
-        self.prefix_documents_cache.clear(writer)
-    }
-
-    pub fn prefix_documents<'txn>(
-        self,
-        reader: &'txn heed::RoTxn<MainT>,
-        prefix: [u8; 4],
-    ) -> ZResult<PrefixDocumentsIter<'txn>> {
-        let start = PrefixKey::new(prefix, 0, 0);
-        let end = PrefixKey::new(prefix, u64::max_value(), u32::max_value());
-        let iter = self.prefix_documents_cache.range(reader, &(start..=end))?;
-        Ok(PrefixDocumentsIter { iter })
-    }
-}
-
-pub struct PrefixDocumentsIter<'txn> {
-    iter: heed::RoRange<'txn, OwnedType<PrefixKey>, CowSlice<Highlight>>,
-}
-
-impl<'txn> Iterator for PrefixDocumentsIter<'txn> {
-    type Item = ZResult<(DocumentId, Cow<'txn, [Highlight]>)>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        match self.iter.next() {
-            Some(Ok((key, highlights))) => {
-                let docid = DocumentId(key.docid.get());
-                Some(Ok((docid, highlights)))
-            }
-            Some(Err(e)) => Some(Err(e)),
-            None => None,
-        }
-    }
-}

meilisearch-core/src/store/prefix_postings_lists_cache.rs

@@ -1,45 +0,0 @@
-use std::borrow::Cow;
-
-use heed::Result as ZResult;
-use heed::types::OwnedType;
-use sdset::{Set, SetBuf};
-use slice_group_by::GroupBy;
-
-use crate::database::MainT;
-use crate::DocIndex;
-use crate::store::{PostingsCodec, Postings};
-
-#[derive(Copy, Clone)]
-pub struct PrefixPostingsListsCache {
-    pub(crate) prefix_postings_lists_cache: heed::Database<OwnedType<[u8; 4]>, PostingsCodec>,
-}
-
-impl PrefixPostingsListsCache {
-    pub fn put_prefix_postings_list(
-        self,
-        writer: &mut heed::RwTxn<MainT>,
-        prefix: [u8; 4],
-        matches: &Set<DocIndex>,
-    ) -> ZResult<()>
-    {
-        let docids = matches.linear_group_by_key(|m| m.document_id).map(|g| g[0].document_id).collect();
-        let docids = Cow::Owned(SetBuf::new_unchecked(docids));
-        let matches = Cow::Borrowed(matches);
-        let postings = Postings { docids, matches };
-
-        self.prefix_postings_lists_cache.put(writer, &prefix, &postings)
-    }
-
-    pub fn clear(self, writer: &mut heed::RwTxn<MainT>) -> ZResult<()> {
-        self.prefix_postings_lists_cache.clear(writer)
-    }
-
-    pub fn prefix_postings_list<'txn>(
-        self,
-        reader: &'txn heed::RoTxn<MainT>,
-        prefix: [u8; 4],
-    ) -> ZResult<Option<Postings<'txn>>>
-    {
-        self.prefix_postings_lists_cache.get(reader, &prefix)
-    }
-}

meilisearch-core/src/store/synonyms.rs

@@ -1,44 +0,0 @@
-use std::borrow::Cow;
-
-use heed::Result as ZResult;
-use heed::types::ByteSlice;
-
-use crate::database::MainT;
-use crate::{FstSetCow, MResult};
-
-#[derive(Copy, Clone)]
-pub struct Synonyms {
-    pub(crate) synonyms: heed::Database<ByteSlice, ByteSlice>,
-}
-
-impl Synonyms {
-    pub fn put_synonyms<A>(self, writer: &mut heed::RwTxn<MainT>, word: &[u8], synonyms: &fst::Set<A>) -> ZResult<()>
-    where A: AsRef<[u8]>,
-    {
-        let bytes = synonyms.as_fst().as_bytes();
-        self.synonyms.put(writer, word, bytes)
-    }
-
-    pub fn del_synonyms(self, writer: &mut heed::RwTxn<MainT>, word: &[u8]) -> ZResult<bool> {
-        self.synonyms.delete(writer, word)
-    }
-
-    pub fn clear(self, writer: &mut heed::RwTxn<MainT>) -> ZResult<()> {
-        self.synonyms.clear(writer)
-    }
-
-    pub(crate) fn synonyms_fst<'txn>(self, reader: &'txn heed::RoTxn<MainT>, word: &[u8]) -> ZResult<FstSetCow<'txn>> {
-        match self.synonyms.get(reader, word)? {
-            Some(bytes) => Ok(fst::Set::new(bytes).unwrap().map_data(Cow::Borrowed).unwrap()),
-            None => Ok(fst::Set::default().map_data(Cow::Owned).unwrap()),
-        }
-    }
-
-    pub fn synonyms(self, reader: &heed::RoTxn<MainT>, word: &[u8]) -> MResult<Vec<String>> {
-        let synonyms = self
-            .synonyms_fst(&reader, word)?
-            .stream()
-            .into_strs()?;
-        Ok(synonyms)
-    }
-}

meilisearch-core/src/store/updates.rs

@@ -1,65 +0,0 @@
-use super::BEU64;
-use crate::database::UpdateT;
-use crate::update::Update;
-use heed::types::{OwnedType, SerdeJson};
-use heed::Result as ZResult;
-
-#[derive(Copy, Clone)]
-pub struct Updates {
-    pub(crate) updates: heed::Database<OwnedType<BEU64>, SerdeJson<Update>>,
-}
-
-impl Updates {
-    // TODO do not trigger deserialize if possible
-    pub fn last_update(self, reader: &heed::RoTxn<UpdateT>) -> ZResult<Option<(u64, Update)>> {
-        match self.updates.last(reader)? {
-            Some((key, data)) => Ok(Some((key.get(), data))),
-            None => Ok(None),
-        }
-    }
-
-    // TODO do not trigger deserialize if possible
-    pub fn first_update(self, reader: &heed::RoTxn<UpdateT>) -> ZResult<Option<(u64, Update)>> {
-        match self.updates.first(reader)? {
-            Some((key, data)) => Ok(Some((key.get(), data))),
-            None => Ok(None),
-        }
-    }
-
-    // TODO do not trigger deserialize if possible
-    pub fn get(self, reader: &heed::RoTxn<UpdateT>, update_id: u64) -> ZResult<Option<Update>> {
-        let update_id = BEU64::new(update_id);
-        self.updates.get(reader, &update_id)
-    }
-
-    pub fn put_update(
-        self,
-        writer: &mut heed::RwTxn<UpdateT>,
-        update_id: u64,
-        update: &Update,
-    ) -> ZResult<()> {
-        // TODO prefer using serde_json?
-        let update_id = BEU64::new(update_id);
-        self.updates.put(writer, &update_id, update)
-    }
-
-    pub fn del_update(self, writer: &mut heed::RwTxn<UpdateT>, update_id: u64) -> ZResult<bool> {
-        let update_id = BEU64::new(update_id);
-        self.updates.delete(writer, &update_id)
-    }
-
-    pub fn pop_front(self, writer: &mut heed::RwTxn<UpdateT>) -> ZResult<Option<(u64, Update)>> {
-        match self.first_update(writer)? {
-            Some((update_id, update)) => {
-                let key = BEU64::new(update_id);
-                self.updates.delete(writer, &key)?;
-                Ok(Some((update_id, update)))
-            }
-            None => Ok(None),
-        }
-    }
-
-    pub fn clear(self, writer: &mut heed::RwTxn<UpdateT>) -> ZResult<()> {
-        self.updates.clear(writer)
-    }
-}

meilisearch-core/src/store/updates_results.rs

@@ -1,45 +0,0 @@
-use super::BEU64;
-use crate::database::UpdateT;
-use crate::update::ProcessedUpdateResult;
-use heed::types::{OwnedType, SerdeJson};
-use heed::Result as ZResult;
-
-#[derive(Copy, Clone)]
-pub struct UpdatesResults {
-    pub(crate) updates_results: heed::Database<OwnedType<BEU64>, SerdeJson<ProcessedUpdateResult>>,
-}
-
-impl UpdatesResults {
-    pub fn last_update(
-        self,
-        reader: &heed::RoTxn<UpdateT>,
-    ) -> ZResult<Option<(u64, ProcessedUpdateResult)>> {
-        match self.updates_results.last(reader)? {
-            Some((key, data)) => Ok(Some((key.get(), data))),
-            None => Ok(None),
-        }
-    }
-
-    pub fn put_update_result(
-        self,
-        writer: &mut heed::RwTxn<UpdateT>,
-        update_id: u64,
-        update_result: &ProcessedUpdateResult,
-    ) -> ZResult<()> {
-        let update_id = BEU64::new(update_id);
-        self.updates_results.put(writer, &update_id, update_result)
-    }
-
-    pub fn update_result(
-        self,
-        reader: &heed::RoTxn<UpdateT>,
-        update_id: u64,
-    ) -> ZResult<Option<ProcessedUpdateResult>> {
-        let update_id = BEU64::new(update_id);
-        self.updates_results.get(reader, &update_id)
-    }
-
-    pub fn clear(self, writer: &mut heed::RwTxn<UpdateT>) -> ZResult<()> {
-        self.updates_results.clear(writer)
-    }
-}

meilisearch-core/src/update/clear_all.rs

@@ -1,34 +0,0 @@
-use crate::database::{MainT, UpdateT};
-use crate::update::{next_update_id, Update};
-use crate::{store, MResult, RankedMap};
-
-pub fn apply_clear_all(
-    writer: &mut heed::RwTxn<MainT>,
-    index: &store::Index,
-) -> MResult<()> {
-    index.main.put_words_fst(writer, &fst::Set::default())?;
-    index.main.put_external_docids(writer, &fst::Map::default())?;
-    index.main.put_internal_docids(writer, &sdset::SetBuf::default())?;
-    index.main.put_ranked_map(writer, &RankedMap::default())?;
-    index.main.put_number_of_documents(writer, |_| 0)?;
-    index.documents_fields.clear(writer)?;
-    index.documents_fields_counts.clear(writer)?;
-    index.postings_lists.clear(writer)?;
-    index.docs_words.clear(writer)?;
-    index.prefix_documents_cache.clear(writer)?;
-    index.prefix_postings_lists_cache.clear(writer)?;
-
-    Ok(())
-}
-
-pub fn push_clear_all(
-    writer: &mut heed::RwTxn<UpdateT>,
-    updates_store: store::Updates,
-    updates_results_store: store::UpdatesResults,
-) -> MResult<u64> {
-    let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
-    let update = Update::clear_all();
-    updates_store.put_update(writer, last_update_id, &update)?;
-
-    Ok(last_update_id)
-}

meilisearch-core/src/update/customs_update.rs

@@ -1,26 +0,0 @@
-
-use crate::database::{MainT, UpdateT};
-use crate::{store, MResult};
-use crate::update::{next_update_id, Update};
-
-pub fn apply_customs_update(
-    writer: &mut heed::RwTxn<MainT>,
-    main_store: store::Main,
-    customs: &[u8],
-) -> MResult<()> {
-    main_store.put_customs(writer, customs)
-}
-
-pub fn push_customs_update(
-    writer: &mut heed::RwTxn<UpdateT>,
-    updates_store: store::Updates,
-    updates_results_store: store::UpdatesResults,
-    customs: Vec<u8>,
-) -> MResult<u64> {
-    let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
-
-    let update = Update::customs(customs);
-    updates_store.put_update(writer, last_update_id, &update)?;
-
-    Ok(last_update_id)
-}

meilisearch-core/src/update/documents_addition.rs

@@ -1,426 +0,0 @@
-use std::borrow::Cow;
-use std::collections::{HashMap, BTreeMap};
-
-use fst::{set::OpBuilder, SetBuilder};
-use indexmap::IndexMap;
-use meilisearch_schema::{Schema, FieldId};
-use meilisearch_types::DocumentId;
-use sdset::{duo::Union, SetOperation};
-use serde::Deserialize;
-use serde_json::Value;
-
-use crate::database::{MainT, UpdateT};
-use crate::database::{UpdateEvent, UpdateEventsEmitter};
-use crate::facets;
-use crate::raw_indexer::RawIndexer;
-use crate::serde::Deserializer;
-use crate::store::{self, DocumentsFields, DocumentsFieldsCounts, DiscoverIds};
-use crate::update::helpers::{index_value, value_to_number, extract_document_id};
-use crate::update::{apply_documents_deletion, compute_short_prefixes, next_update_id, Update};
-use crate::{Error, MResult, RankedMap};
-
-pub struct DocumentsAddition<D> {
-    updates_store: store::Updates,
-    updates_results_store: store::UpdatesResults,
-    updates_notifier: UpdateEventsEmitter,
-    documents: Vec<D>,
-    is_partial: bool,
-}
-
-impl<D> DocumentsAddition<D> {
-    pub fn new(
-        updates_store: store::Updates,
-        updates_results_store: store::UpdatesResults,
-        updates_notifier: UpdateEventsEmitter,
-    ) -> DocumentsAddition<D> {
-        DocumentsAddition {
-            updates_store,
-            updates_results_store,
-            updates_notifier,
-            documents: Vec::new(),
-            is_partial: false,
-        }
-    }
-
-    pub fn new_partial(
-        updates_store: store::Updates,
-        updates_results_store: store::UpdatesResults,
-        updates_notifier: UpdateEventsEmitter,
-    ) -> DocumentsAddition<D> {
-        DocumentsAddition {
-            updates_store,
-            updates_results_store,
-            updates_notifier,
-            documents: Vec::new(),
-            is_partial: true,
-        }
-    }
-
-    pub fn update_document(&mut self, document: D) {
-        self.documents.push(document);
-    }
-
-    pub fn finalize(self, writer: &mut heed::RwTxn<UpdateT>) -> MResult<u64>
-    where
-        D: serde::Serialize,
-    {
-        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
-        let update_id = push_documents_addition(
-            writer,
-            self.updates_store,
-            self.updates_results_store,
-            self.documents,
-            self.is_partial,
-        )?;
-        Ok(update_id)
-    }
-}
-
-impl<D> Extend<D> for DocumentsAddition<D> {
-    fn extend<T: IntoIterator<Item = D>>(&mut self, iter: T) {
-        self.documents.extend(iter)
-    }
-}
-
-pub fn push_documents_addition<D: serde::Serialize>(
-    writer: &mut heed::RwTxn<UpdateT>,
-    updates_store: store::Updates,
-    updates_results_store: store::UpdatesResults,
-    addition: Vec<D>,
-    is_partial: bool,
-) -> MResult<u64> {
-    let mut values = Vec::with_capacity(addition.len());
-    for add in addition {
-        let vec = serde_json::to_vec(&add)?;
-        let add = serde_json::from_slice(&vec)?;
-        values.push(add);
-    }
-
-    let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
-
-    let update = if is_partial {
-        Update::documents_partial(values)
-    } else {
-        Update::documents_addition(values)
-    };
-
-    updates_store.put_update(writer, last_update_id, &update)?;
-
-    Ok(last_update_id)
-}
-
-#[allow(clippy::too_many_arguments)]
-fn index_document<A>(
-    writer: &mut heed::RwTxn<MainT>,
-    documents_fields: DocumentsFields,
-    documents_fields_counts: DocumentsFieldsCounts,
-    ranked_map: &mut RankedMap,
-    indexer: &mut RawIndexer<A>,
-    schema: &Schema,
-    field_id: FieldId,
-    document_id: DocumentId,
-    value: &Value,
-) -> MResult<()>
-where A: AsRef<[u8]>,
-{
-    let serialized = serde_json::to_vec(value)?;
-    documents_fields.put_document_field(writer, document_id, field_id, &serialized)?;
-
-    if let Some(indexed_pos) = schema.is_indexed(field_id) {
-        let number_of_words = index_value(indexer, document_id, *indexed_pos, value);
-        if let Some(number_of_words) = number_of_words {
-            documents_fields_counts.put_document_field_count(
-                writer,
-                document_id,
-                *indexed_pos,
-                number_of_words as u16,
-            )?;
-        }
-    }
-
-    if schema.is_ranked(field_id) {
-        let number = value_to_number(value).unwrap_or_default();
-        ranked_map.insert(document_id, field_id, number);
-    }
-
-    Ok(())
-}
-
-pub fn apply_addition<'a, 'b>(
-    writer: &'a mut heed::RwTxn<'b, MainT>,
-    index: &store::Index,
-    new_documents: Vec<IndexMap<String, Value>>,
-    partial: bool
-) -> MResult<()>
-{
-    let mut schema = match index.main.schema(writer)? {
-        Some(schema) => schema,
-        None => return Err(Error::SchemaMissing),
-    };
-
-    // Retrieve the documents ids related structures
-    let external_docids = index.main.external_docids(writer)?;
-    let internal_docids = index.main.internal_docids(writer)?;
-    let mut available_ids = DiscoverIds::new(&internal_docids);
-
-    let primary_key = schema.primary_key().ok_or(Error::MissingPrimaryKey)?;
-
-    // 1. store documents ids for future deletion
-    let mut documents_additions = HashMap::new();
-    let mut new_external_docids = BTreeMap::new();
-    let mut new_internal_docids = Vec::with_capacity(new_documents.len());
-
-    for mut document in new_documents {
-        let external_docids_get = |docid: &str| {
-            match (external_docids.get(docid), new_external_docids.get(docid)) {
-                (_, Some(&id))
-                | (Some(id), _) => Some(id as u32),
-                (None, None) => None,
-            }
-        };
-
-        let (internal_docid, external_docid) =
-            extract_document_id(
-                &primary_key,
-                &document,
-                &external_docids_get,
-                &mut available_ids,
-            )?;
-
-        new_external_docids.insert(external_docid, internal_docid.0 as u64);
-        new_internal_docids.push(internal_docid);
-
-        if partial {
-            let mut deserializer = Deserializer {
-                document_id: internal_docid,
-                reader: writer,
-                documents_fields: index.documents_fields,
-                schema: &schema,
-                fields: None,
-            };
-
-            let old_document = Option::<HashMap<String, Value>>::deserialize(&mut deserializer)?;
-            if let Some(old_document) = old_document {
-                for (key, value) in old_document {
-                    document.entry(key).or_insert(value);
-                }
-            }
-        }
-        documents_additions.insert(internal_docid, document);
-    }
-
-    // 2. remove the documents postings lists
-    let number_of_inserted_documents = documents_additions.len();
-    let documents_ids = new_external_docids.iter().map(|(id, _)| id.clone()).collect();
-    apply_documents_deletion(writer, index, documents_ids)?;
-
-    let mut ranked_map = match index.main.ranked_map(writer)? {
-        Some(ranked_map) => ranked_map,
-        None => RankedMap::default(),
-    };
-
-    let stop_words = index.main.stop_words_fst(writer)?.map_data(Cow::into_owned)?;
-
-
-    let mut indexer = RawIndexer::new(stop_words);
-
-    // For each document in this update
-    for (document_id, document) in &documents_additions {
-        // For each key-value pair in the document.
-        for (attribute, value) in document {
-            let field_id = schema.insert_and_index(&attribute)?;
-            index_document(
-                writer,
-                index.documents_fields,
-                index.documents_fields_counts,
-                &mut ranked_map,
-                &mut indexer,
-                &schema,
-                field_id,
-                *document_id,
-                &value,
-            )?;
-        }
-    }
-
-    write_documents_addition_index(
-        writer,
-        index,
-        &ranked_map,
-        number_of_inserted_documents,
-        indexer,
-    )?;
-
-    index.main.put_schema(writer, &schema)?;
-
-    let new_external_docids = fst::Map::from_iter(new_external_docids.iter().map(|(ext, id)| (ext, *id as u64)))?;
-    let new_internal_docids = sdset::SetBuf::from_dirty(new_internal_docids);
-    index.main.merge_external_docids(writer, &new_external_docids)?;
-    index.main.merge_internal_docids(writer, &new_internal_docids)?;
-
-    // recompute all facet attributes after document update.
-    if let Some(attributes_for_facetting) = index.main.attributes_for_faceting(writer)? {
-        let docids = index.main.internal_docids(writer)?;
-        let facet_map = facets::facet_map_from_docids(writer, index, &docids, attributes_for_facetting.as_ref())?;
-        index.facets.add(writer, facet_map)?;
-    }
-
-    // update is finished; update sorted document id cache with new state
-    let mut document_ids = index.main.internal_docids(writer)?.to_vec();
-    super::cache_document_ids_sorted(writer, &ranked_map, index, &mut document_ids)?;
-
-    Ok(())
-}
-
-pub fn apply_documents_partial_addition<'a, 'b>(
-    writer: &'a mut heed::RwTxn<'b, MainT>,
-    index: &store::Index,
-    new_documents: Vec<IndexMap<String, Value>>,
-) -> MResult<()> {
-    apply_addition(writer, index, new_documents, true)
-}
-
-pub fn apply_documents_addition<'a, 'b>(
-    writer: &'a mut heed::RwTxn<'b, MainT>,
-    index: &store::Index,
-    new_documents: Vec<IndexMap<String, Value>>,
-) -> MResult<()> {
-    apply_addition(writer, index, new_documents, false)
-}
-
-pub fn reindex_all_documents(writer: &mut heed::RwTxn<MainT>, index: &store::Index) -> MResult<()> {
-    let schema = match index.main.schema(writer)? {
-        Some(schema) => schema,
-        None => return Err(Error::SchemaMissing),
-    };
-
-    let mut ranked_map = RankedMap::default();
-
-    // 1. retrieve all documents ids
-    let mut documents_ids_to_reindex = Vec::new();
-    for result in index.documents_fields_counts.documents_ids(writer)? {
-        let document_id = result?;
-        documents_ids_to_reindex.push(document_id);
-    }
-
-    // 2. remove the documents posting lists
-    index.main.put_words_fst(writer, &fst::Set::default())?;
-    index.main.put_ranked_map(writer, &ranked_map)?;
-    index.main.put_number_of_documents(writer, |_| 0)?;
-    index.facets.clear(writer)?;
-    index.postings_lists.clear(writer)?;
-    index.docs_words.clear(writer)?;
-
-    let stop_words = index.main
-        .stop_words_fst(writer)?
-        .map_data(Cow::into_owned)
-        .unwrap();
-
-    let number_of_inserted_documents = documents_ids_to_reindex.len();
-    let mut indexer = RawIndexer::new(stop_words);
-    let mut ram_store = HashMap::new();
-
-    if let Some(ref attributes_for_facetting) = index.main.attributes_for_faceting(writer)? {
-        let facet_map = facets::facet_map_from_docids(writer, &index, &documents_ids_to_reindex, &attributes_for_facetting)?;
-        index.facets.add(writer, facet_map)?;
-    }
-    // ^-- https://github.com/meilisearch/MeiliSearch/pull/631#issuecomment-626624470 --v
-    for document_id in &documents_ids_to_reindex {
-        for result in index.documents_fields.document_fields(writer, *document_id)? {
-            let (field_id, bytes) = result?;
-            let value: Value = serde_json::from_slice(bytes)?;
-            ram_store.insert((document_id, field_id), value);
-        }
-
-        // For each key-value pair in the document.
-        for ((document_id, field_id), value) in ram_store.drain() {
-            index_document(
-                writer,
-                index.documents_fields,
-                index.documents_fields_counts,
-                &mut ranked_map,
-                &mut indexer,
-                &schema,
-                field_id,
-                *document_id,
-                &value,
-            )?;
-        }
-    }
-
-    // 4. write the new index in the main store
-    write_documents_addition_index(
-        writer,
-        index,
-        &ranked_map,
-        number_of_inserted_documents,
-        indexer,
-    )?;
-
-    index.main.put_schema(writer, &schema)?;
-
-    // recompute all facet attributes after document update.
-    if let Some(attributes_for_facetting) = index.main.attributes_for_faceting(writer)? {
-        let docids = index.main.internal_docids(writer)?;
-        let facet_map = facets::facet_map_from_docids(writer, index, &docids, attributes_for_facetting.as_ref())?;
-        index.facets.add(writer, facet_map)?;
-    }
-
-    // update is finished; update sorted document id cache with new state
-    let mut document_ids = index.main.internal_docids(writer)?.to_vec();
-    super::cache_document_ids_sorted(writer, &ranked_map, index, &mut document_ids)?;
-
-    Ok(())
-}
-
-pub fn write_documents_addition_index<A>(
-    writer: &mut heed::RwTxn<MainT>,
-    index: &store::Index,
-    ranked_map: &RankedMap,
-    number_of_inserted_documents: usize,
-    indexer: RawIndexer<A>,
-) -> MResult<()>
-where A: AsRef<[u8]>,
-{
-    let indexed = indexer.build();
-    let mut delta_words_builder = SetBuilder::memory();
-
-    for (word, delta_set) in indexed.words_doc_indexes {
-        delta_words_builder.insert(&word).unwrap();
-
-        let set = match index.postings_lists.postings_list(writer, &word)? {
-            Some(postings) => Union::new(&postings.matches, &delta_set).into_set_buf(),
-            None => delta_set,
-        };
-
-        index.postings_lists.put_postings_list(writer, &word, &set)?;
-    }
-
-    for (id, words) in indexed.docs_words {
-        index.docs_words.put_doc_words(writer, id, &words)?;
-    }
-
-    let delta_words = delta_words_builder.into_set();
-
-    let words_fst = index.main.words_fst(writer)?;
-    let words = if !words_fst.is_empty() {
-        let op = OpBuilder::new()
-            .add(words_fst.stream())
-            .add(delta_words.stream())
-            .r#union();
-
-        let mut words_builder = SetBuilder::memory();
-        words_builder.extend_stream(op).unwrap();
-        words_builder.into_set()
-    } else {
-        delta_words
-    };
-
-    index.main.put_words_fst(writer, &words)?;
-    index.main.put_ranked_map(writer, ranked_map)?;
-    index.main.put_number_of_documents(writer, |old| old + number_of_inserted_documents as u64)?;
-
-    compute_short_prefixes(writer, &words, index)?;
-
-    Ok(())
-}