Bump meilidb-core to v0.7.0 and meilidb-http to v0.2.0

Prefer using a global update callback common to all indexes

.gitignore

@@ -1,5 +1,4 @@
 /target
-Cargo.lock
 **/*.csv
 **/*.json_lines
 **/*.rs.bk

Cargo.toml

@@ -4,6 +4,7 @@ members = [
     "meilidb-http",
     "meilidb-schema",
     "meilidb-tokenizer",
+    "meilidb-types",
 ]
 
 [profile.release]

LICENSE

@@ -1,13 +1,21 @@
-“Commons Clause” License Condition v1.0
+MIT License
 
-The Software is provided to you by the Licensor under the License, as defined below, subject to the following condition.
+Copyright (c) [year] [fullname]
 
-Without limiting other conditions in the License, the grant of rights under the License will not include, and the License does not grant to you, the right to Sell the Software.
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
 
-For purposes of the foregoing, “Sell” means practicing any or all of the rights granted to you under the License to provide to third parties, for a fee or other consideration (including without limitation fees for hosting or consulting/ support services related to the Software), a product or service whose value derives, entirely or substantially, from the functionality of the Software. Any license notice or attribution required by the License must also include this Commons Clause License Condition notice.
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
 
-Software: MeiliDB
-
-License: MIT
-
-Licensor: MEILI SAS
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -4,7 +4,9 @@
 [![dependency status](https://deps.rs/repo/github/meilisearch/MeiliDB/status.svg)](https://deps.rs/repo/github/meilisearch/MeiliDB)
 [![License](https://img.shields.io/badge/license-commons%20clause-lightgrey)](https://commonsclause.com/)
 
-A _full-text search database_ based on the fast [LMDB key-value store](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database).
+Ultra relevant and instant full-text search API.
+
+MeiliSearch is a powerful, fast, open-source, easy to use and deploy search engine. The search and indexation are fully customizable and handles features like typo-tolerance, filters, and ranking.
 
 ## Features
 
@@ -27,16 +29,91 @@ You can [read the deep dive](deep-dive.md) if you want more information on the e
 
 We will be proud if you submit issues and pull requests. You can help to grow this project and start contributing by checking [issues tagged "good-first-issue"](https://github.com/meilisearch/MeiliDB/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22). It is a good start!
 
-The project is only a library yet. It means that there is no binary provided yet. To get started, you can check the examples wich are made to work with the data located in the `datasets/` folder.
+[![crates.io demo gif](misc/crates-io-demo.gif)](https://crates.meilisearch.com)
 
-MeiliDB will be a binary in a near future so you will be able to use it as a database out-of-the-box. We should be able to query it using HTTP. This is our current goal, [see the milestones](https://github.com/meilisearch/MeiliDB/milestones). In the end, the binary will be a bunch of network protocols and wrappers around the library - which will also be published on [crates.io](https://crates.io). Both the binary and the library will follow the same update cycle.
+> Meili helps the Rust community find crates on [crates.meilisearch.com](https://crates.meilisearch.com)
+
+
+
+## Quick Start
+
+You can deploy your own instant, relevant and typo-tolerant MeiliDB search engine by yourself too.
+Something similar to the demo above can be achieve by following these little three steps first.
+You will need to create your own web front display to make it pretty though.
+
+### Deploy the Server
+
+If you have not installed Rust and its package manager `cargo` yet, go to [the installation page](https://www.rust-lang.org/tools/install).<br/>
+You can deploy the server on your own machine, it will listen to HTTP requests on the 8080 port by default.
+
+```bash
+rustup override set nightly
+cargo run --release
+```
+
+For more logs during the execution, run:
+```bash
+RUST_LOG=info cargo run --release
+```
+
+### Create an Index and Upload Some Documents
+
+MeiliDB can serve multiple indexes, with different kinds of documents,
+therefore, it is required to create the index before sending documents to it.
+
+```bash
+curl -i -X POST 'http://127.0.0.1:8080/indexes/movies'
+```
+
+Now that the server knows about our brand new index, we can send it data.
+We provided you a little dataset, it is available in the `datasets/` directory.
+
+```bash
+curl -i -X POST 'http://127.0.0.1:8080/indexes/movies/documents' \
+  --header 'content-type: application/json' \
+  --data @datasets/movies/movies.json
+```
+
+### Search for Documents
+
+The search engine is now aware of our documents and can serve those via our HTTP server again.
+The [`jq` command line tool](https://stedolan.github.io/jq/) can greatly help you read the server responses.
+
+```bash
+curl 'http://127.0.0.1:8080/indexes/movies/search?q=botman'
+```
+
+```json
+{
+  "hits": [
+    {
+      "id": "29751",
+      "title": "Batman Unmasked: The Psychology of the Dark Knight",
+      "poster": "https://image.tmdb.org/t/p/w1280/jjHu128XLARc2k4cJrblAvZe0HE.jpg",
+      "overview": "Delve into the world of Batman and the vigilante justice tha",
+      "release_date": "2008-07-15"
+    },
+    {
+      "id": "471474",
+      "title": "Batman: Gotham by Gaslight",
+      "poster": "https://image.tmdb.org/t/p/w1280/7souLi5zqQCnpZVghaXv0Wowi0y.jpg",
+      "overview": "ve Victorian Age Gotham City, Batman begins his war on crime",
+      "release_date": "2018-01-12"
+    }
+  ],
+  "offset": 0,
+  "limit": 2,
+  "processingTimeMs": 1,
+  "query": "botman"
+}
+```
 
 
 
 ## Performances
 
-With a database composed of _100 353_ documents with _352_ attributes each and _3_ of them indexed.
-So more than _300 000_ fields indexed for _35 million_ stored we can handle more than _2.8k req/sec_ with an average response time of _9 ms_ on an Intel i7-7700 (8) @ 4.2GHz.
+With a dataset composed of _100 353_ documents with _352_ attributes each and _3_ of them indexed.
+So more than _300 000_ fields indexed for _35 million_ stored we can handle more than _2.8k req/sec_ with an average response time of _9 ms_ on an Intel i7-7700 (8) @ 4.2GHz.
 
 Requests are made using [wrk](https://github.com/wg/wrk) and scripted to simulate real users queries.
 
@@ -51,20 +128,24 @@ Requests/sec:   2806.46
 Transfer/sec:    759.17KB
 ```
 
+We also indexed a dataset containing something like _12 millions_ cities names in _24 minutes_ on a machine with _8 cores_, _64 GB of RAM_ and a _300 GB NMVe_ SSD.<br/>
+The resulting database was _16 GB_ and search results were between _30 ms_ and _4 seconds_ for short prefix queries.
+
 ### Notes
 
 With Rust 1.32 the allocator has been [changed to use the system allocator](https://blog.rust-lang.org/2019/01/17/Rust-1.32.0.html#jemalloc-is-removed-by-default).
 We have seen much better performances when [using jemalloc as the global allocator](https://github.com/alexcrichton/jemallocator#documentation).
 
-## Usage and examples
+## Usage and Examples
 
-Currently MeiliDB do not provide an http server but you can run the example binary.
+MeiliDB also provides an example binary that is mostly used for features testing.
+Notice that the example binary is faster to index data as it does read direct CSV files and not JSON HTTP payloads.
 
 The _index_ subcommand has been made to create an index and inject documents into it. Using the command line below, the index will be named _movies_ and the _19 700_ movies of the `datasets/` will be injected in MeiliDB.
 
 ```bash
 cargo run --release --example from_file -- \
-    index example.mdb datasets/movies/data.csv \
+    index example.mdb datasets/movies/movies.csv \
     --schema datasets/movies/schema.toml
 ```
 
@@ -72,8 +153,8 @@ Once the first command is done, you can query the freshly created _movies_ index
 
 ```bash
 cargo run --release --example from_file -- \
-    search example.mdb
-    --number 4 \
+    search example.mdb \
+    --number-results 4 \
     --filter '!adult' \
     id popularity adult original_title
 ```

azure-pipelines.yml

@@ -42,11 +42,11 @@ jobs:
         displayName: 'Build MeiliDB'
       - task: CopyFiles@2
         inputs:
-          contents: '$(System.DefaultWorkingDirectory)/target/release/libmeilidb.rlib'
+          contents: '$(System.DefaultWorkingDirectory)/target/release/meilidb-http'
           targetFolder: $(Build.ArtifactStagingDirectory)
         displayName: 'Copy build'
       - task: PublishBuildArtifacts@1
         inputs:
-          artifactName: libmeilidb.rlib
+          artifactName: meilidb
         displayName: 'Upload artifacts'
 

meilidb-core/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "meilidb-core"
-version = "0.6.1"
+version = "0.7.0"
 authors = ["Kerollmops <clement@meilisearch.com>"]
 edition = "2018"
 
@@ -8,14 +8,18 @@ edition = "2018"
 arc-swap = "0.4.3"
 bincode = "1.1.4"
 byteorder = "1.3.2"
-crossbeam-channel = "0.3.9"
+chrono = { version = "0.4.9", features = ["serde"] }
+crossbeam-channel = "0.4.0"
 deunicode = "1.0.0"
 env_logger = "0.7.0"
+fst = { version = "0.3.5", default-features = false }
 hashbrown = { version = "0.6.0", features = ["serde"] }
 heed = "0.5.0"
+levenshtein_automata = { version = "0.1.1", features = ["fst_automaton"] }
 log = "0.4.8"
 meilidb-schema = { path = "../meilidb-schema", version = "0.6.0" }
 meilidb-tokenizer = { path = "../meilidb-tokenizer", version = "0.6.0" }
+meilidb-types = { path = "../meilidb-types", version = "0.1.0" }
 once_cell = "1.2.0"
 ordered-float = { version = "1.0.2", features = ["serde"] }
 sdset = "0.3.3"
@@ -25,15 +29,6 @@ siphasher = "0.3.0"
 slice-group-by = "0.2.6"
 zerocopy = "0.2.8"
 
-[dependencies.levenshtein_automata]
-git = "https://github.com/Kerollmops/levenshtein-automata.git"
-branch = "arc-byte-slice"
-features = ["fst_automaton"]
-
-[dependencies.fst]
-git = "https://github.com/Kerollmops/fst.git"
-branch = "arc-byte-slice"
-
 [dev-dependencies]
 assert_matches = "1.3"
 csv = "1.0.7"

meilidb-core/examples/from_file.rs

@@ -104,14 +104,14 @@ fn index_command(command: IndexCommand, database: Database) -> Result<(), Box<dy
     let start = Instant::now();
 
     let (sender, receiver) = mpsc::sync_channel(100);
-    let update_fn = move |update: ProcessedUpdateResult| sender.send(update.update_id).unwrap();
+    let update_fn =
+        move |_name: &str, update: ProcessedUpdateResult| sender.send(update.update_id).unwrap();
     let index = match database.open_index(&command.index_name) {
         Some(index) => index,
         None => database.create_index(&command.index_name).unwrap(),
     };
 
-    let done = database.set_update_callback(&command.index_name, Box::new(update_fn));
-    assert!(done, "could not set the index update function");
+    database.set_update_callback(Box::new(update_fn));
 
     let env = &database.env;
 
@@ -195,8 +195,9 @@ fn index_command(command: IndexCommand, database: Database) -> Result<(), Box<dy
     );
 
     if let Some(path) = command.compact_to_path {
+        fs::create_dir_all(&path)?;
         let start = Instant::now();
-        let _file = database.copy_and_compact_to_path(&path)?;
+        let _file = database.copy_and_compact_to_path(path.join("data.mdb"))?;
         println!(
             "database compacted in {:.2?} at: {:?}",
             start.elapsed(),
@@ -217,7 +218,11 @@ fn display_highlights(text: &str, ranges: &[usize]) -> io::Result<()> {
             _ => unreachable!(),
         };
         if highlighted {
-            stdout.set_color(ColorSpec::new().set_fg(Some(Color::Yellow)))?;
+            stdout.set_color(
+                ColorSpec::new()
+                    .set_fg(Some(Color::Yellow))
+                    .set_underline(true),
+            )?;
         }
         write!(&mut stdout, "{}", &text[start..end])?;
         stdout.reset()?;

meilidb-core/src/database.rs

@@ -4,21 +4,22 @@ use std::path::Path;
 use std::sync::{Arc, RwLock};
 use std::{fs, thread};
 
-use crossbeam_channel::Receiver;
+use crossbeam_channel::{Receiver, Sender};
 use heed::types::{Str, Unit};
 use heed::{CompactionOption, Result as ZResult};
 use log::debug;
 
 use crate::{store, update, Index, MResult};
 
-pub type BoxUpdateFn = Box<dyn Fn(update::ProcessedUpdateResult) + Send + Sync + 'static>;
+pub type BoxUpdateFn = Box<dyn Fn(&str, update::ProcessedUpdateResult) + Send + Sync + 'static>;
 type ArcSwapFn = arc_swap::ArcSwapOption<BoxUpdateFn>;
 
 pub struct Database {
     pub env: heed::Env,
     common_store: heed::PolyDatabase,
     indexes_store: heed::Database<Str, Unit>,
-    indexes: RwLock<HashMap<String, (Index, Arc<ArcSwapFn>, thread::JoinHandle<()>)>>,
+    indexes: RwLock<HashMap<String, (Index, thread::JoinHandle<()>)>>,
+    update_fn: Arc<ArcSwapFn>,
 }
 
 macro_rules! r#break_try {
@@ -33,9 +34,23 @@ macro_rules! r#break_try {
     };
 }
 
-fn update_awaiter(receiver: Receiver<()>, env: heed::Env, update_fn: Arc<ArcSwapFn>, index: Index) {
-    for () in receiver {
-        // consume all updates in order (oldest first)
+pub enum UpdateEvent {
+    NewUpdate,
+    MustStop,
+}
+
+pub type UpdateEvents = Receiver<UpdateEvent>;
+pub type UpdateEventsEmitter = Sender<UpdateEvent>;
+
+fn update_awaiter(
+    receiver: UpdateEvents,
+    env: heed::Env,
+    index_name: &str,
+    update_fn: Arc<ArcSwapFn>,
+    index: Index,
+) {
+    let mut receiver = receiver.into_iter();
+    while let Some(UpdateEvent::NewUpdate) = receiver.next() {
         loop {
             // instantiate a main/parent transaction
             let mut writer = break_try!(env.write_txn(), "LMDB write transaction begin failed");
@@ -60,7 +75,7 @@ fn update_awaiter(receiver: Receiver<()>, env: heed::Env, update_fn: Arc<ArcSwap
             let status = break_try!(result, "update task failed");
 
             // commit the nested transaction if the update was successful, abort it otherwise
-            if status.result.is_ok() {
+            if status.error.is_none() {
                 break_try!(nested_writer.commit(), "commit nested transaction failed");
             } else {
                 nested_writer.abort()
@@ -76,10 +91,12 @@ fn update_awaiter(receiver: Receiver<()>, env: heed::Env, update_fn: Arc<ArcSwap
 
             // call the user callback when the update and the result are written consistently
             if let Some(ref callback) = *update_fn.load() {
-                (callback)(status);
+                (callback)(index_name, status);
             }
         }
     }
+
+    debug!("update loop system stopped");
 }
 
 impl Database {
@@ -93,6 +110,7 @@ impl Database {
 
         let common_store = env.create_poly_database(Some("common"))?;
         let indexes_store = env.create_database::<Str, Unit>(Some("indexes"))?;
+        let update_fn = Arc::new(ArcSwapFn::empty());
 
         // list all indexes that needs to be opened
         let mut must_open = Vec::new();
@@ -118,21 +136,27 @@ impl Database {
                     continue;
                 }
             };
-            let update_fn = Arc::new(ArcSwapFn::empty());
 
             let env_clone = env.clone();
             let index_clone = index.clone();
+            let name_clone = index_name.clone();
             let update_fn_clone = update_fn.clone();
 
             let handle = thread::spawn(move || {
-                update_awaiter(receiver, env_clone, update_fn_clone, index_clone)
+                update_awaiter(
+                    receiver,
+                    env_clone,
+                    &name_clone,
+                    update_fn_clone,
+                    index_clone,
+                )
             });
 
             // send an update notification to make sure that
             // possible pre-boot updates are consumed
-            sender.send(()).unwrap();
+            sender.send(UpdateEvent::NewUpdate).unwrap();
 
-            let result = indexes.insert(index_name, (index, update_fn, handle));
+            let result = indexes.insert(index_name, (index, handle));
             assert!(
                 result.is_none(),
                 "The index should not have been already open"
@@ -144,6 +168,7 @@ impl Database {
             common_store,
             indexes_store,
             indexes: RwLock::new(indexes),
+            update_fn,
         })
     }
 
@@ -170,43 +195,56 @@ impl Database {
 
                 let env_clone = self.env.clone();
                 let index_clone = index.clone();
-
-                let no_update_fn = Arc::new(ArcSwapFn::empty());
-                let no_update_fn_clone = no_update_fn.clone();
+                let name_clone = name.to_owned();
+                let update_fn_clone = self.update_fn.clone();
 
                 let handle = thread::spawn(move || {
-                    update_awaiter(receiver, env_clone, no_update_fn_clone, index_clone)
+                    update_awaiter(
+                        receiver,
+                        env_clone,
+                        &name_clone,
+                        update_fn_clone,
+                        index_clone,
+                    )
                 });
 
                 writer.commit()?;
-                entry.insert((index.clone(), no_update_fn, handle));
+                entry.insert((index.clone(), handle));
 
                 Ok(index)
             }
         }
     }
 
-    pub fn set_update_callback(&self, name: impl AsRef<str>, update_fn: BoxUpdateFn) -> bool {
-        let indexes_lock = self.indexes.read().unwrap();
-        match indexes_lock.get(name.as_ref()) {
-            Some((_, current_update_fn, _)) => {
-                let update_fn = Some(Arc::new(update_fn));
-                current_update_fn.swap(update_fn);
-                true
+    pub fn delete_index(&self, name: impl AsRef<str>) -> MResult<bool> {
+        let name = name.as_ref();
+        let mut indexes_lock = self.indexes.write().unwrap();
+
+        match indexes_lock.remove_entry(name) {
+            Some((name, (index, handle))) => {
+                // remove the index name from the list of indexes
+                // and clear all the LMDB dbi
+                let mut writer = self.env.write_txn()?;
+                self.indexes_store.delete(&mut writer, &name)?;
+                store::clear(&mut writer, &index)?;
+                writer.commit()?;
+
+                // join the update loop thread to ensure it is stopped
+                handle.join().unwrap();
+
+                Ok(true)
             }
-            None => false,
+            None => Ok(false),
         }
     }
 
-    pub fn unset_update_callback(&self, name: impl AsRef<str>) -> bool {
-        let indexes_lock = self.indexes.read().unwrap();
-        match indexes_lock.get(name.as_ref()) {
-            Some((_, current_update_fn, _)) => {
-                current_update_fn.swap(None);
-                true
-            }
-            None => false,
-        }
+    pub fn set_update_callback(&self, update_fn: BoxUpdateFn) {
+        let update_fn = Some(Arc::new(update_fn));
+        self.update_fn.swap(update_fn);
+    }
+
+    pub fn unset_update_callback(&self) {
+        self.update_fn.swap(None);
     }
 
     pub fn copy_and_compact_to_path<P: AsRef<Path>>(&self, path: P) -> ZResult<File> {
@@ -226,7 +264,10 @@ impl Database {
 #[cfg(test)]
 mod tests {
     use super::*;
+
     use crate::update::{ProcessedUpdateResult, UpdateStatus};
+    use crate::DocumentId;
+    use serde::de::IgnoredAny;
     use std::sync::mpsc;
 
     #[test]
@@ -237,11 +278,12 @@ mod tests {
         let env = &database.env;
 
         let (sender, receiver) = mpsc::sync_channel(100);
-        let update_fn = move |update: ProcessedUpdateResult| sender.send(update.update_id).unwrap();
+        let update_fn = move |_name: &str, update: ProcessedUpdateResult| {
+            sender.send(update.update_id).unwrap()
+        };
         let index = database.create_index("test").unwrap();
 
-        let done = database.set_update_callback("test", Box::new(update_fn));
-        assert!(done, "could not set the index update function");
+        database.set_update_callback(Box::new(update_fn));
 
         let schema = {
             let data = r#"
@@ -260,8 +302,6 @@ mod tests {
 
         let mut writer = env.write_txn().unwrap();
         let _update_id = index.schema_update(&mut writer, schema).unwrap();
-
-        // don't forget to commit...
         writer.commit().unwrap();
 
         let mut additions = index.documents_addition();
@@ -283,8 +323,6 @@ mod tests {
 
         let mut writer = env.write_txn().unwrap();
         let update_id = additions.finalize(&mut writer).unwrap();
-
-        // don't forget to commit...
         writer.commit().unwrap();
 
         // block until the transaction is processed
@@ -292,7 +330,7 @@ mod tests {
 
         let reader = env.read_txn().unwrap();
         let result = index.update_status(&reader, update_id).unwrap();
-        assert_matches!(result, UpdateStatus::Processed(status) if status.result.is_ok());
+        assert_matches!(result, Some(UpdateStatus::Processed { content }) if content.error.is_none());
     }
 
     #[test]
@@ -303,11 +341,12 @@ mod tests {
         let env = &database.env;
 
         let (sender, receiver) = mpsc::sync_channel(100);
-        let update_fn = move |update: ProcessedUpdateResult| sender.send(update.update_id).unwrap();
+        let update_fn = move |_name: &str, update: ProcessedUpdateResult| {
+            sender.send(update.update_id).unwrap()
+        };
         let index = database.create_index("test").unwrap();
 
-        let done = database.set_update_callback("test", Box::new(update_fn));
-        assert!(done, "could not set the index update function");
+        database.set_update_callback(Box::new(update_fn));
 
         let schema = {
             let data = r#"
@@ -326,8 +365,6 @@ mod tests {
 
         let mut writer = env.write_txn().unwrap();
         let _update_id = index.schema_update(&mut writer, schema).unwrap();
-
-        // don't forget to commit...
         writer.commit().unwrap();
 
         let mut additions = index.documents_addition();
@@ -348,8 +385,6 @@ mod tests {
 
         let mut writer = env.write_txn().unwrap();
         let update_id = additions.finalize(&mut writer).unwrap();
-
-        // don't forget to commit...
         writer.commit().unwrap();
 
         // block until the transaction is processed
@@ -357,6 +392,453 @@ mod tests {
 
         let reader = env.read_txn().unwrap();
         let result = index.update_status(&reader, update_id).unwrap();
-        assert_matches!(result, UpdateStatus::Processed(status) if status.result.is_err());
+        assert_matches!(result, Some(UpdateStatus::Processed { content }) if content.error.is_some());
+    }
+
+    #[test]
+    fn ignored_words_too_long() {
+        let dir = tempfile::tempdir().unwrap();
+
+        let database = Database::open_or_create(dir.path()).unwrap();
+        let env = &database.env;
+
+        let (sender, receiver) = mpsc::sync_channel(100);
+        let update_fn = move |_name: &str, update: ProcessedUpdateResult| {
+            sender.send(update.update_id).unwrap()
+        };
+        let index = database.create_index("test").unwrap();
+
+        database.set_update_callback(Box::new(update_fn));
+
+        let schema = {
+            let data = r#"
+                identifier = "id"
+
+                [attributes."name"]
+                displayed = true
+                indexed = true
+            "#;
+            toml::from_str(data).unwrap()
+        };
+
+        let mut writer = env.write_txn().unwrap();
+        let _update_id = index.schema_update(&mut writer, schema).unwrap();
+        writer.commit().unwrap();
+
+        let mut additions = index.documents_addition();
+
+        let doc1 = serde_json::json!({
+            "id": 123,
+            "name": "s̷̡̢̡̧̺̜̞͕͉͉͕̜͔̟̼̥̝͍̟̖͔͔̪͉̲̹̝̣̖͎̞̤̥͓͎̭̩͕̙̩̿̀̋̅̈́̌́̏̍̄̽͂̆̾̀̿̕̚̚͜͠͠ͅͅļ̵̨̨̨̰̦̻̳̖̳͚̬̫͚̦͖͈̲̫̣̩̥̻̙̦̱̼̠̖̻̼̘̖͉̪̜̠̙͖̙̩͔̖̯̩̲̿̽͋̔̿̍̓͂̍̿͊͆̃͗̔̎͐͌̾̆͗́̆̒̔̾̅̚̚͜͜ͅͅī̵̛̦̅̔̓͂͌̾́͂͛̎̋͐͆̽̂̋̋́̾̀̉̓̏̽́̑̀͒̇͋͛̈́̃̉̏͊̌̄̽̿̏̇͘̕̚̕p̶̧̛̛̖̯̗͕̝̗̭̱͙̖̗̟̟̐͆̊̂͐̋̓̂̈́̓͊̆͌̾̾͐͋͗͌̆̿̅͆̈́̈́̉͋̍͊͗̌̓̅̈̎̇̃̎̈́̉̐̋͑̃͘̕͘d̴̢̨̛͕̘̯͖̭̮̝̝̐̊̈̅̐̀͒̀́̈́̀͌̽͛͆͑̀̽̿͛̃̋̇̎̀́̂́͘͠͝ǫ̵̨̛̮̩̘͚̬̯̖̱͍̼͑͑̓̐́̑̿̈́̔͌̂̄͐͝ģ̶̧̜͇̣̭̺̪̺̖̻͖̮̭̣̙̻͒͊͗̓̓͒̀̀ͅ",
+        });
+
+        additions.update_document(doc1);
+
+        let mut writer = env.write_txn().unwrap();
+        let update_id = additions.finalize(&mut writer).unwrap();
+        writer.commit().unwrap();
+
+        // block until the transaction is processed
+        let _ = receiver.into_iter().find(|id| *id == update_id);
+
+        let reader = env.read_txn().unwrap();
+        let result = index.update_status(&reader, update_id).unwrap();
+        assert_matches!(result, Some(UpdateStatus::Processed { content }) if content.error.is_none());
+    }
+
+    #[test]
+    fn add_schema_attributes_at_end() {
+        let dir = tempfile::tempdir().unwrap();
+
+        let database = Database::open_or_create(dir.path()).unwrap();
+        let env = &database.env;
+
+        let (sender, receiver) = mpsc::sync_channel(100);
+        let update_fn = move |_name: &str, update: ProcessedUpdateResult| {
+            sender.send(update.update_id).unwrap()
+        };
+        let index = database.create_index("test").unwrap();
+
+        database.set_update_callback(Box::new(update_fn));
+
+        let schema = {
+            let data = r#"
+                identifier = "id"
+
+                [attributes."name"]
+                displayed = true
+                indexed = true
+
+                [attributes."description"]
+                displayed = true
+                indexed = true
+            "#;
+            toml::from_str(data).unwrap()
+        };
+
+        let mut writer = env.write_txn().unwrap();
+        let _update_id = index.schema_update(&mut writer, schema).unwrap();
+        writer.commit().unwrap();
+
+        let mut additions = index.documents_addition();
+
+        let doc1 = serde_json::json!({
+            "id": 123,
+            "name": "Marvin",
+            "description": "My name is Marvin",
+        });
+
+        let doc2 = serde_json::json!({
+            "id": 234,
+            "name": "Kevin",
+            "description": "My name is Kevin",
+        });
+
+        additions.update_document(doc1);
+        additions.update_document(doc2);
+
+        let mut writer = env.write_txn().unwrap();
+        let _update_id = additions.finalize(&mut writer).unwrap();
+        writer.commit().unwrap();
+
+        let schema = {
+            let data = r#"
+                identifier = "id"
+
+                [attributes."name"]
+                displayed = true
+                indexed = true
+
+                [attributes."description"]
+                displayed = true
+                indexed = true
+
+                [attributes."age"]
+                displayed = true
+                indexed = true
+
+                [attributes."sex"]
+                displayed = true
+                indexed = true
+            "#;
+            toml::from_str(data).unwrap()
+        };
+
+        let mut writer = env.write_txn().unwrap();
+        let update_id = index.schema_update(&mut writer, schema).unwrap();
+        writer.commit().unwrap();
+
+        // block until the transaction is processed
+        let _ = receiver.iter().find(|id| *id == update_id);
+
+        // check if it has been accepted
+        let reader = env.read_txn().unwrap();
+        let result = index.update_status(&reader, update_id).unwrap();
+        assert_matches!(result, Some(UpdateStatus::Processed { content }) if content.error.is_none());
+        reader.abort();
+
+        let mut additions = index.documents_addition();
+
+        let doc1 = serde_json::json!({
+            "id": 123,
+            "name": "Marvin",
+            "description": "My name is Marvin",
+            "age": 21,
+            "sex": "Male",
+        });
+
+        let doc2 = serde_json::json!({
+            "id": 234,
+            "name": "Kevin",
+            "description": "My name is Kevin",
+            "age": 23,
+            "sex": "Male",
+        });
+
+        additions.update_document(doc1);
+        additions.update_document(doc2);
+
+        let mut writer = env.write_txn().unwrap();
+        let update_id = additions.finalize(&mut writer).unwrap();
+        writer.commit().unwrap();
+
+        // block until the transaction is processed
+        let _ = receiver.iter().find(|id| *id == update_id);
+
+        // check if it has been accepted
+        let reader = env.read_txn().unwrap();
+        let result = index.update_status(&reader, update_id).unwrap();
+        assert_matches!(result, Some(UpdateStatus::Processed { content }) if content.error.is_none());
+
+        // even try to search for a document
+        let results = index.query_builder().query(&reader, "21 ", 0..20).unwrap();
+        assert_matches!(results.len(), 1);
+
+        reader.abort();
+
+        // try to introduce attributes in the middle of the schema
+        let schema = {
+            let data = r#"
+                identifier = "id"
+
+                [attributes."name"]
+                displayed = true
+                indexed = true
+
+                [attributes."description"]
+                displayed = true
+                indexed = true
+
+                [attributes."city"]
+                displayed = true
+                indexed = true
+
+                [attributes."age"]
+                displayed = true
+                indexed = true
+
+                [attributes."sex"]
+                displayed = true
+                indexed = true
+            "#;
+            toml::from_str(data).unwrap()
+        };
+
+        let mut writer = env.write_txn().unwrap();
+        let update_id = index.schema_update(&mut writer, schema).unwrap();
+        writer.commit().unwrap();
+
+        // block until the transaction is processed
+        let _ = receiver.iter().find(|id| *id == update_id);
+
+        // check if it has been accepted
+        let reader = env.read_txn().unwrap();
+        let result = index.update_status(&reader, update_id).unwrap();
+        assert_matches!(result, Some(UpdateStatus::Processed { content }) if content.error.is_some());
+    }
+
+    #[test]
+    fn deserialize_documents() {
+        let dir = tempfile::tempdir().unwrap();
+
+        let database = Database::open_or_create(dir.path()).unwrap();
+        let env = &database.env;
+
+        let (sender, receiver) = mpsc::sync_channel(100);
+        let update_fn = move |_name: &str, update: ProcessedUpdateResult| {
+            sender.send(update.update_id).unwrap()
+        };
+        let index = database.create_index("test").unwrap();
+
+        database.set_update_callback(Box::new(update_fn));
+
+        let schema = {
+            let data = r#"
+                identifier = "id"
+
+                [attributes."name"]
+                displayed = true
+                indexed = true
+
+                [attributes."description"]
+                displayed = true
+                indexed = true
+            "#;
+            toml::from_str(data).unwrap()
+        };
+
+        let mut writer = env.write_txn().unwrap();
+        let _update_id = index.schema_update(&mut writer, schema).unwrap();
+        writer.commit().unwrap();
+
+        let mut additions = index.documents_addition();
+
+        // DocumentId(7900334843754999545)
+        let doc1 = serde_json::json!({
+            "id": 123,
+            "name": "Marvin",
+            "description": "My name is Marvin",
+        });
+
+        // DocumentId(8367468610878465872)
+        let doc2 = serde_json::json!({
+            "id": 234,
+            "name": "Kevin",
+            "description": "My name is Kevin",
+        });
+
+        additions.update_document(doc1);
+        additions.update_document(doc2);
+
+        let mut writer = env.write_txn().unwrap();
+        let update_id = additions.finalize(&mut writer).unwrap();
+        writer.commit().unwrap();
+
+        // block until the transaction is processed
+        let _ = receiver.into_iter().find(|id| *id == update_id);
+
+        let reader = env.read_txn().unwrap();
+        let result = index.update_status(&reader, update_id).unwrap();
+        assert_matches!(result, Some(UpdateStatus::Processed { content }) if content.error.is_none());
+
+        let document: Option<IgnoredAny> = index.document(&reader, None, DocumentId(25)).unwrap();
+        assert!(document.is_none());
+
+        let document: Option<IgnoredAny> = index
+            .document(&reader, None, DocumentId(7900334843754999545))
+            .unwrap();
+        assert!(document.is_some());
+
+        let document: Option<IgnoredAny> = index
+            .document(&reader, None, DocumentId(8367468610878465872))
+            .unwrap();
+        assert!(document.is_some());
+    }
+
+    #[test]
+    fn partial_document_update() {
+        let dir = tempfile::tempdir().unwrap();
+
+        let database = Database::open_or_create(dir.path()).unwrap();
+        let env = &database.env;
+
+        let (sender, receiver) = mpsc::sync_channel(100);
+        let update_fn = move |_name: &str, update: ProcessedUpdateResult| {
+            sender.send(update.update_id).unwrap()
+        };
+        let index = database.create_index("test").unwrap();
+
+        database.set_update_callback(Box::new(update_fn));
+
+        let schema = {
+            let data = r#"
+                identifier = "id"
+
+                [attributes."id"]
+                displayed = true
+
+                [attributes."name"]
+                displayed = true
+                indexed = true
+
+                [attributes."description"]
+                displayed = true
+                indexed = true
+            "#;
+            toml::from_str(data).unwrap()
+        };
+
+        let mut writer = env.write_txn().unwrap();
+        let _update_id = index.schema_update(&mut writer, schema).unwrap();
+        writer.commit().unwrap();
+
+        let mut additions = index.documents_addition();
+
+        // DocumentId(7900334843754999545)
+        let doc1 = serde_json::json!({
+            "id": 123,
+            "name": "Marvin",
+            "description": "My name is Marvin",
+        });
+
+        // DocumentId(8367468610878465872)
+        let doc2 = serde_json::json!({
+            "id": 234,
+            "name": "Kevin",
+            "description": "My name is Kevin",
+        });
+
+        additions.update_document(doc1);
+        additions.update_document(doc2);
+
+        let mut writer = env.write_txn().unwrap();
+        let update_id = additions.finalize(&mut writer).unwrap();
+        writer.commit().unwrap();
+
+        // block until the transaction is processed
+        let _ = receiver.iter().find(|id| *id == update_id);
+
+        let reader = env.read_txn().unwrap();
+        let result = index.update_status(&reader, update_id).unwrap();
+        assert_matches!(result, Some(UpdateStatus::Processed { content }) if content.error.is_none());
+
+        let document: Option<IgnoredAny> = index.document(&reader, None, DocumentId(25)).unwrap();
+        assert!(document.is_none());
+
+        let document: Option<IgnoredAny> = index
+            .document(&reader, None, DocumentId(7900334843754999545))
+            .unwrap();
+        assert!(document.is_some());
+
+        let document: Option<IgnoredAny> = index
+            .document(&reader, None, DocumentId(8367468610878465872))
+            .unwrap();
+        assert!(document.is_some());
+
+        reader.abort();
+
+        let mut partial_additions = index.documents_partial_addition();
+
+        // DocumentId(7900334843754999545)
+        let partial_doc1 = serde_json::json!({
+            "id": 123,
+            "description": "I am the new Marvin",
+        });
+
+        // DocumentId(8367468610878465872)
+        let partial_doc2 = serde_json::json!({
+            "id": 234,
+            "description": "I am the new Kevin",
+        });
+
+        partial_additions.update_document(partial_doc1);
+        partial_additions.update_document(partial_doc2);
+
+        let mut writer = env.write_txn().unwrap();
+        let update_id = partial_additions.finalize(&mut writer).unwrap();
+        writer.commit().unwrap();
+
+        // block until the transaction is processed
+        let _ = receiver.iter().find(|id| *id == update_id);
+
+        let reader = env.read_txn().unwrap();
+        let result = index.update_status(&reader, update_id).unwrap();
+        assert_matches!(result, Some(UpdateStatus::Processed { content }) if content.error.is_none());
+
+        let document: Option<serde_json::Value> = index
+            .document(&reader, None, DocumentId(7900334843754999545))
+            .unwrap();
+
+        let new_doc1 = serde_json::json!({
+            "id": 123,
+            "name": "Marvin",
+            "description": "I am the new Marvin",
+        });
+        assert_eq!(document, Some(new_doc1));
+
+        let document: Option<serde_json::Value> = index
+            .document(&reader, None, DocumentId(8367468610878465872))
+            .unwrap();
+
+        let new_doc2 = serde_json::json!({
+            "id": 234,
+            "name": "Kevin",
+            "description": "I am the new Kevin",
+        });
+        assert_eq!(document, Some(new_doc2));
+    }
+
+    #[test]
+    fn delete_index() {
+        let dir = tempfile::tempdir().unwrap();
+
+        let database = Database::open_or_create(dir.path()).unwrap();
+        let _index = database.create_index("test").unwrap();
+
+        let deleted = database.delete_index("test").unwrap();
+        assert!(deleted);
+
+        let result = database.open_index("test");
+        assert!(result.is_none());
     }
 }

meilidb-core/src/error.rs

@@ -97,7 +97,7 @@ pub enum UnsupportedOperation {
     SchemaAlreadyExists,
     CannotUpdateSchemaIdentifier,
     CannotReorderSchemaAttribute,
-    CannotIntroduceNewSchemaAttribute,
+    CanOnlyIntroduceNewSchemaAttributesAtEnd,
     CannotRemoveSchemaAttribute,
 }
 
@@ -108,8 +108,8 @@ impl fmt::Display for UnsupportedOperation {
             SchemaAlreadyExists => write!(f, "Cannot update index which already have a schema"),
             CannotUpdateSchemaIdentifier => write!(f, "Cannot update the identifier of a schema"),
             CannotReorderSchemaAttribute => write!(f, "Cannot reorder the attributes of a schema"),
-            CannotIntroduceNewSchemaAttribute => {
-                write!(f, "Cannot introduce new attributes in a schema")
+            CanOnlyIntroduceNewSchemaAttributesAtEnd => {
+                write!(f, "Can only introduce new attributes at end of a schema")
             }
             CannotRemoveSchemaAttribute => write!(f, "Cannot remove attributes from a schema"),
         }

meilidb-core/src/lib.rs

@@ -25,79 +25,7 @@ pub use self::ranked_map::RankedMap;
 pub use self::raw_document::RawDocument;
 pub use self::store::Index;
 pub use self::update::{EnqueuedUpdateResult, ProcessedUpdateResult, UpdateStatus, UpdateType};
-
-use ::serde::{Deserialize, Serialize};
-use zerocopy::{AsBytes, FromBytes};
-
-/// Represent an internally generated document unique identifier.
-///
-/// It is used to inform the database the document you want to deserialize.
-/// Helpful for custom ranking.
-#[derive(
-    Debug,
-    Copy,
-    Clone,
-    Eq,
-    PartialEq,
-    PartialOrd,
-    Ord,
-    Hash,
-    Serialize,
-    Deserialize,
-    AsBytes,
-    FromBytes,
-)]
-#[repr(C)]
-pub struct DocumentId(pub u64);
-
-/// This structure represent the position of a word
-/// in a document and its attributes.
-///
-/// This is stored in the map, generated at index time,
-/// extracted and interpreted at search time.
-#[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, AsBytes, FromBytes)]
-#[repr(C)]
-pub struct DocIndex {
-    /// The document identifier where the word was found.
-    pub document_id: DocumentId,
-
-    /// The attribute in the document where the word was found
-    /// along with the index in it.
-    pub attribute: u16,
-    pub word_index: u16,
-
-    /// The position in bytes where the word was found
-    /// along with the length of it.
-    ///
-    /// It informs on the original word area in the text indexed
-    /// without needing to run the tokenizer again.
-    pub char_index: u16,
-    pub char_length: u16,
-}
-
-/// This structure represent a matching word with informations
-/// on the location of the word in the document.
-///
-/// The order of the field is important because it defines
-/// the way these structures are ordered between themselves.
-#[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
-pub struct Highlight {
-    /// The attribute in the document where the word was found
-    /// along with the index in it.
-    pub attribute: u16,
-
-    /// The position in bytes where the word was found.
-    ///
-    /// It informs on the original word area in the text indexed
-    /// without needing to run the tokenizer again.
-    pub char_index: u16,
-
-    /// The length in bytes of the found word.
-    ///
-    /// It informs on the original word area in the text indexed
-    /// without needing to run the tokenizer again.
-    pub char_length: u16,
-}
+pub use meilidb_types::{DocIndex, DocumentId, Highlight};
 
 #[doc(hidden)]
 #[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]

meilidb-core/src/query_builder.rs

@@ -1,9 +1,9 @@
 use hashbrown::HashMap;
 use std::convert::TryFrom;
-use std::mem;
 use std::ops::Range;
 use std::rc::Rc;
 use std::time::{Duration, Instant};
+use std::{cmp, mem};
 
 use fst::{IntoStreamer, Streamer};
 use sdset::SetBuf;
@@ -178,7 +178,7 @@ fn fetch_raw_documents(
                 let distance = dfa.eval(input).to_u8();
                 let is_exact = *is_exact && distance == 0 && input.len() == *query_len;
 
-                let covered_area = if query.len() > input.len() {
+                let covered_area = if *query_len > input.len() {
                     input.len()
                 } else {
                     prefix_damerau_levenshtein(query.as_bytes(), input).1
@@ -202,10 +202,13 @@ fn fetch_raw_documents(
                             is_exact,
                         };
 
+                        let covered_area = u16::try_from(covered_area).unwrap_or(u16::max_value());
+                        let covered_area = cmp::min(covered_area, di.char_length);
+
                         let highlight = Highlight {
                             attribute: di.attribute,
                             char_index: di.char_index,
-                            char_length: u16::try_from(covered_area).unwrap_or(u16::max_value()),
+                            char_length: covered_area,
                         };
 
                         tmp_matches.push((di.document_id, id, match_, highlight));

meilidb-core/src/raw_indexer.rs

@@ -7,6 +7,8 @@ use meilidb_schema::SchemaAttr;
 use meilidb_tokenizer::{is_cjk, SeqTokenizer, Token, Tokenizer};
 use sdset::SetBuf;
 
+const WORD_LENGTH_LIMIT: usize = 80;
+
 type Word = Vec<u8>; // TODO make it be a SmallVec
 
 pub struct RawIndexer {
@@ -37,54 +39,8 @@ impl RawIndexer {
 
     pub fn index_text(&mut self, id: DocumentId, attr: SchemaAttr, text: &str) -> usize {
         let mut number_of_words = 0;
-        let lowercase_text = text.to_lowercase();
-        let deunicoded = deunicode_with_tofu(&lowercase_text, "");
 
-        // TODO compute the deunicoded version after the cjk check
-        let next = if !lowercase_text.contains(is_cjk) && lowercase_text != deunicoded {
-            Some(deunicoded)
-        } else {
-            None
-        };
-        let iter = Some(lowercase_text).into_iter().chain(next);
-
-        for text in iter {
-            // we must not count 2 times the same words
-            number_of_words = 0;
-
-            for token in Tokenizer::new(&text) {
-                let must_continue = index_token(
-                    token,
-                    id,
-                    attr,
-                    self.word_limit,
-                    &self.stop_words,
-                    &mut self.words_doc_indexes,
-                    &mut self.docs_words,
-                );
-
-                if !must_continue {
-                    break;
-                }
-
-                number_of_words += 1;
-            }
-        }
-
-        number_of_words
-    }
-
-    pub fn index_text_seq<'a, I, IT>(&mut self, id: DocumentId, attr: SchemaAttr, iter: I)
-    where
-        I: IntoIterator<Item = &'a str, IntoIter = IT>,
-        IT: Iterator<Item = &'a str> + Clone,
-    {
-        // TODO serialize this to one call to the SeqTokenizer loop
-
-        let lowercased: Vec<_> = iter.into_iter().map(str::to_lowercase).collect();
-        let iter = lowercased.iter().map(|t| t.as_str());
-
-        for token in SeqTokenizer::new(iter) {
+        for token in Tokenizer::new(text) {
             let must_continue = index_token(
                 token,
                 id,
@@ -95,27 +51,21 @@ impl RawIndexer {
                 &mut self.docs_words,
             );
 
+            number_of_words += 1;
+
             if !must_continue {
                 break;
             }
         }
 
-        let deunicoded: Vec<_> = lowercased
-            .into_iter()
-            .map(|lowercase_text| {
-                if lowercase_text.contains(is_cjk) {
-                    return lowercase_text;
-                }
-                let deunicoded = deunicode_with_tofu(&lowercase_text, "");
-                if lowercase_text != deunicoded {
-                    deunicoded
-                } else {
-                    lowercase_text
-                }
-            })
-            .collect();
-        let iter = deunicoded.iter().map(|t| t.as_str());
+        number_of_words
+    }
 
+    pub fn index_text_seq<'a, I>(&mut self, id: DocumentId, attr: SchemaAttr, iter: I)
+    where
+        I: IntoIterator<Item = &'a str>,
+    {
+        let iter = iter.into_iter();
         for token in SeqTokenizer::new(iter) {
             let must_continue = index_token(
                 token,
@@ -170,15 +120,38 @@ fn index_token(
         return false;
     }
 
+    let lower = token.word.to_lowercase();
+    let token = Token {
+        word: &lower,
+        ..token
+    };
+
     if !stop_words.contains(&token.word) {
         match token_to_docindex(id, attr, token) {
             Some(docindex) => {
                 let word = Vec::from(token.word);
-                words_doc_indexes
-                    .entry(word.clone())
-                    .or_insert_with(Vec::new)
-                    .push(docindex);
-                docs_words.entry(id).or_insert_with(Vec::new).push(word);
+
+                if word.len() <= WORD_LENGTH_LIMIT {
+                    words_doc_indexes
+                        .entry(word.clone())
+                        .or_insert_with(Vec::new)
+                        .push(docindex);
+                    docs_words.entry(id).or_insert_with(Vec::new).push(word);
+
+                    if !lower.contains(is_cjk) {
+                        let unidecoded = deunicode_with_tofu(&lower, "");
+                        if unidecoded != lower && !unidecoded.is_empty() {
+                            let word = Vec::from(unidecoded);
+                            if word.len() <= WORD_LENGTH_LIMIT {
+                                words_doc_indexes
+                                    .entry(word.clone())
+                                    .or_insert_with(Vec::new)
+                                    .push(docindex);
+                                docs_words.entry(id).or_insert_with(Vec::new).push(word);
+                            }
+                        }
+                    }
+                }
             }
             None => return false,
         }
@@ -224,10 +197,8 @@ mod tests {
         assert!(words_doc_indexes.get(&b"aspirateur"[..]).is_some());
         assert!(words_doc_indexes.get(&b"ai"[..]).is_some());
         assert!(words_doc_indexes.get(&b"eteindre"[..]).is_some());
-
-        // with the ugly apostrophe...
         assert!(words_doc_indexes
-            .get(&"l’éteindre".to_owned().into_bytes())
+            .get(&"éteindre".to_owned().into_bytes())
             .is_some());
     }
 
@@ -248,10 +219,8 @@ mod tests {
         assert!(words_doc_indexes.get(&b"aspirateur"[..]).is_some());
         assert!(words_doc_indexes.get(&b"ai"[..]).is_some());
         assert!(words_doc_indexes.get(&b"eteindre"[..]).is_some());
-
-        // with the ugly apostrophe...
         assert!(words_doc_indexes
-            .get(&"l’éteindre".to_owned().into_bytes())
+            .get(&"éteindre".to_owned().into_bytes())
             .is_some());
     }
 
@@ -277,10 +246,26 @@ mod tests {
         assert!(words_doc_indexes.get(&b"ai"[..]).is_none());
         assert!(words_doc_indexes.get(&b"de"[..]).is_none());
         assert!(words_doc_indexes.get(&b"eteindre"[..]).is_some());
-
-        // with the ugly apostrophe...
         assert!(words_doc_indexes
-            .get(&"l’éteindre".to_owned().into_bytes())
+            .get(&"éteindre".to_owned().into_bytes())
+            .is_some());
+    }
+
+    #[test]
+    fn no_empty_unidecode() {
+        let mut indexer = RawIndexer::new(fst::Set::default());
+
+        let docid = DocumentId(0);
+        let attr = SchemaAttr(0);
+        let text = "🇯🇵";
+        indexer.index_text(docid, attr, text);
+
+        let Indexed {
+            words_doc_indexes, ..
+        } = indexer.build();
+
+        assert!(words_doc_indexes
+            .get(&"🇯🇵".to_owned().into_bytes())
             .is_some());
     }
 }

meilidb-core/src/serde/deserializer.rs

@@ -63,13 +63,14 @@ impl<'de, 'a, 'b> de::Deserializer<'de> for &'b mut Deserializer<'a> {
     where
         V: de::Visitor<'de>,
     {
-        self.deserialize_map(visitor)
+        self.deserialize_option(visitor)
     }
 
-    forward_to_deserialize_any! {
-        bool i8 i16 i32 i64 i128 u8 u16 u32 u64 u128 f32 f64 char str string
-        bytes byte_buf option unit unit_struct newtype_struct seq tuple
-        tuple_struct struct enum identifier ignored_any
+    fn deserialize_option<V>(self, visitor: V) -> Result<V::Value, Self::Error>
+    where
+        V: de::Visitor<'de>,
+    {
+        self.deserialize_map(visitor)
     }
 
     fn deserialize_map<V>(self, visitor: V) -> Result<V::Value, Self::Error>
@@ -104,16 +105,29 @@ impl<'de, 'a, 'b> de::Deserializer<'de> for &'b mut Deserializer<'a> {
                 }
             });
 
-        let map_deserializer = de::value::MapDeserializer::new(iter);
-        let result = visitor
-            .visit_map(map_deserializer)
-            .map_err(DeserializerError::from);
+        let mut iter = iter.peekable();
+
+        let result = match iter.peek() {
+            Some(_) => {
+                let map_deserializer = de::value::MapDeserializer::new(iter);
+                visitor
+                    .visit_some(map_deserializer)
+                    .map_err(DeserializerError::from)
+            }
+            None => visitor.visit_none(),
+        };
 
         match error.take() {
             Some(error) => Err(error.into()),
             None => result,
         }
     }
+
+    forward_to_deserialize_any! {
+        bool i8 i16 i32 i64 i128 u8 u16 u32 u64 u128 f32 f64 char str string
+        bytes byte_buf unit unit_struct newtype_struct seq tuple
+        tuple_struct struct enum identifier ignored_any
+    }
 }
 
 struct Value(SerdeJsonDeserializer<SerdeJsonIoRead<Cursor<Vec<u8>>>>);

meilidb-core/src/store/docs_words.rs

@@ -39,7 +39,7 @@ impl DocsWords {
         match self.docs_words.get(reader, &document_id)? {
             Some(bytes) => {
                 let len = bytes.len();
-                let bytes = Arc::from(bytes);
+                let bytes = Arc::new(bytes.to_owned());
                 let fst = fst::raw::Fst::from_shared_bytes(bytes, 0, len).unwrap();
                 Ok(Some(fst::Set::from(fst)))
             }

meilidb-core/src/store/main.rs

@@ -18,6 +18,10 @@ pub struct Main {
 }
 
 impl Main {
+    pub fn clear(self, writer: &mut heed::RwTxn) -> ZResult<()> {
+        self.main.clear(writer)
+    }
+
     pub fn put_words_fst(self, writer: &mut heed::RwTxn, fst: &fst::Set) -> ZResult<()> {
         let bytes = fst.as_fst().as_bytes();
         self.main.put::<Str, ByteSlice>(writer, WORDS_KEY, bytes)
@@ -27,7 +31,7 @@ impl Main {
         match self.main.get::<Str, ByteSlice>(reader, WORDS_KEY)? {
             Some(bytes) => {
                 let len = bytes.len();
-                let bytes = Arc::from(bytes);
+                let bytes = Arc::new(bytes.to_owned());
                 let fst = fst::raw::Fst::from_shared_bytes(bytes, 0, len).unwrap();
                 Ok(Some(fst::Set::from(fst)))
             }
@@ -64,7 +68,7 @@ impl Main {
         match self.main.get::<Str, ByteSlice>(reader, SYNONYMS_KEY)? {
             Some(bytes) => {
                 let len = bytes.len();
-                let bytes = Arc::from(bytes);
+                let bytes = Arc::new(bytes.to_owned());
                 let fst = fst::raw::Fst::from_shared_bytes(bytes, 0, len).unwrap();
                 Ok(Some(fst::Set::from(fst)))
             }
@@ -82,7 +86,7 @@ impl Main {
         match self.main.get::<Str, ByteSlice>(reader, STOP_WORDS_KEY)? {
             Some(bytes) => {
                 let len = bytes.len();
-                let bytes = Arc::from(bytes);
+                let bytes = Arc::new(bytes.to_owned());
                 let fst = fst::raw::Fst::from_shared_bytes(bytes, 0, len).unwrap();
                 Ok(Some(fst::Set::from(fst)))
             }

meilidb-core/src/store/mod.rs

@@ -22,10 +22,11 @@ use std::collections::HashSet;
 
 use heed::Result as ZResult;
 use meilidb_schema::{Schema, SchemaAttr};
-use serde::de;
+use serde::de::{self, Deserialize};
 use zerocopy::{AsBytes, FromBytes};
 
 use crate::criterion::Criteria;
+use crate::database::{UpdateEvent, UpdateEventsEmitter};
 use crate::serde::Deserializer;
 use crate::{query_builder::QueryBuilder, update, DocumentId, Error, MResult};
 
@@ -91,7 +92,7 @@ pub struct Index {
 
     pub updates: Updates,
     pub updates_results: UpdatesResults,
-    updates_notifier: crossbeam_channel::Sender<()>,
+    updates_notifier: UpdateEventsEmitter,
 }
 
 impl Index {
@@ -120,9 +121,7 @@ impl Index {
             attributes: attributes.as_ref(),
         };
 
-        // TODO: currently we return an error if all document fields are missing,
-        //       returning None would have been better
-        Ok(T::deserialize(&mut deserializer).map(Some)?)
+        Ok(Option::<T>::deserialize(&mut deserializer)?)
     }
 
     pub fn document_attribute<T: de::DeserializeOwned>(
@@ -141,12 +140,12 @@ impl Index {
     }
 
     pub fn schema_update(&self, writer: &mut heed::RwTxn, schema: Schema) -> MResult<u64> {
-        let _ = self.updates_notifier.send(());
+        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
         update::push_schema_update(writer, self.updates, self.updates_results, schema)
     }
 
     pub fn customs_update(&self, writer: &mut heed::RwTxn, customs: Vec<u8>) -> ZResult<u64> {
-        let _ = self.updates_notifier.send(());
+        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
         update::push_customs_update(writer, self.updates, self.updates_results, customs)
     }
 
@@ -158,6 +157,14 @@ impl Index {
         )
     }
 
+    pub fn documents_partial_addition<D>(&self) -> update::DocumentsAddition<D> {
+        update::DocumentsAddition::new_partial(
+            self.updates,
+            self.updates_results,
+            self.updates_notifier.clone(),
+        )
+    }
+
     pub fn documents_deletion(&self) -> update::DocumentsDeletion {
         update::DocumentsDeletion::new(
             self.updates,
@@ -167,7 +174,7 @@ impl Index {
     }
 
     pub fn clear_all(&self, writer: &mut heed::RwTxn) -> MResult<u64> {
-        let _ = self.updates_notifier.send(());
+        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
         update::push_clear_all(writer, self.updates, self.updates_results)
     }
 
@@ -214,7 +221,7 @@ impl Index {
         &self,
         reader: &heed::RoTxn,
         update_id: u64,
-    ) -> MResult<update::UpdateStatus> {
+    ) -> MResult<Option<update::UpdateStatus>> {
         update::update_status(reader, self.updates, self.updates_results, update_id)
     }
 
@@ -227,17 +234,19 @@ impl Index {
             updates.reserve(last_id as usize);
 
             for id in 0..=last_id {
-                let update = self.update_status(reader, id)?;
-                updates.push(update);
-                last_update_result_id = id;
+                if let Some(update) = self.update_status(reader, id)? {
+                    updates.push(update);
+                    last_update_result_id = id;
+                }
             }
         }
 
         // retrieve all enqueued updates
         if let Some((last_id, _)) = self.updates.last_update_id(reader)? {
-            for id in last_update_result_id + 1..last_id {
-                let update = self.update_status(reader, id)?;
-                updates.push(update);
+            for id in last_update_result_id + 1..=last_id {
+                if let Some(update) = self.update_status(reader, id)? {
+                    updates.push(update);
+                }
             }
         }
 
@@ -270,7 +279,7 @@ impl Index {
 pub fn create(
     env: &heed::Env,
     name: &str,
-    updates_notifier: crossbeam_channel::Sender<()>,
+    updates_notifier: UpdateEventsEmitter,
 ) -> MResult<Index> {
     // create all the store names
     let main_name = main_name(name);
@@ -310,7 +319,7 @@ pub fn create(
 pub fn open(
     env: &heed::Env,
     name: &str,
-    updates_notifier: crossbeam_channel::Sender<()>,
+    updates_notifier: UpdateEventsEmitter,
 ) -> MResult<Option<Index>> {
     // create all the store names
     let main_name = main_name(name);
@@ -370,3 +379,19 @@ pub fn open(
         updates_notifier,
     }))
 }
+
+pub fn clear(writer: &mut heed::RwTxn, index: &Index) -> MResult<()> {
+    // send a stop event to the update loop of the index
+    index.updates_notifier.send(UpdateEvent::MustStop).unwrap();
+
+    // clear all the stores
+    index.main.clear(writer)?;
+    index.postings_lists.clear(writer)?;
+    index.documents_fields.clear(writer)?;
+    index.documents_fields_counts.clear(writer)?;
+    index.synonyms.clear(writer)?;
+    index.docs_words.clear(writer)?;
+    index.updates.clear(writer)?;
+    index.updates_results.clear(writer)?;
+    Ok(())
+}

meilidb-core/src/store/synonyms.rs

@@ -22,11 +22,15 @@ impl Synonyms {
         self.synonyms.delete(writer, word)
     }
 
+    pub fn clear(self, writer: &mut heed::RwTxn) -> ZResult<()> {
+        self.synonyms.clear(writer)
+    }
+
     pub fn synonyms(self, reader: &heed::RoTxn, word: &[u8]) -> ZResult<Option<fst::Set>> {
         match self.synonyms.get(reader, word)? {
             Some(bytes) => {
                 let len = bytes.len();
-                let bytes = Arc::from(bytes);
+                let bytes = Arc::new(bytes.to_owned());
                 let fst = fst::raw::Fst::from_shared_bytes(bytes, 0, len).unwrap();
                 Ok(Some(fst::Set::from(fst)))
             }

meilidb-core/src/store/updates.rs

@@ -52,4 +52,8 @@ impl Updates {
             None => Ok(None),
         }
     }
+
+    pub fn clear(self, writer: &mut heed::RwTxn) -> ZResult<()> {
+        self.updates.clear(writer)
+    }
 }

meilidb-core/src/store/updates_results.rs

@@ -1,12 +1,11 @@
 use super::BEU64;
 use crate::update::ProcessedUpdateResult;
-use heed::types::{OwnedType, SerdeBincode};
+use heed::types::{OwnedType, SerdeJson};
 use heed::Result as ZResult;
 
 #[derive(Copy, Clone)]
 pub struct UpdatesResults {
-    pub(crate) updates_results:
-        heed::Database<OwnedType<BEU64>, SerdeBincode<ProcessedUpdateResult>>,
+    pub(crate) updates_results: heed::Database<OwnedType<BEU64>, SerdeJson<ProcessedUpdateResult>>,
 }
 
 impl UpdatesResults {
@@ -38,4 +37,8 @@ impl UpdatesResults {
         let update_id = BEU64::new(update_id);
         self.updates_results.get(reader, &update_id)
     }
+
+    pub fn clear(self, writer: &mut heed::RwTxn) -> ZResult<()> {
+        self.updates_results.clear(writer)
+    }
 }

meilidb-core/src/update/clear_all.rs

@@ -26,7 +26,7 @@ pub fn push_clear_all(
     updates_results_store: store::UpdatesResults,
 ) -> MResult<u64> {
     let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
-    let update = Update::ClearAll;
+    let update = Update::clear_all();
     updates_store.put_update(writer, last_update_id, &update)?;
 
     Ok(last_update_id)

meilidb-core/src/update/customs_update.rs

@@ -18,7 +18,7 @@ pub fn push_customs_update(
 ) -> ZResult<u64> {
     let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
 
-    let update = Update::Customs(customs);
+    let update = Update::customs(customs);
     updates_store.put_update(writer, last_update_id, &update)?;
 
     Ok(last_update_id)

meilidb-core/src/update/documents_addition.rs

@@ -2,10 +2,11 @@ use std::collections::HashMap;
 
 use fst::{set::OpBuilder, SetBuilder};
 use sdset::{duo::Union, SetOperation};
-use serde::Serialize;
+use serde::{Deserialize, Serialize};
 
+use crate::database::{UpdateEvent, UpdateEventsEmitter};
 use crate::raw_indexer::RawIndexer;
-use crate::serde::{extract_document_id, serialize_value, Serializer};
+use crate::serde::{extract_document_id, serialize_value, Deserializer, Serializer};
 use crate::store;
 use crate::update::{apply_documents_deletion, next_update_id, Update};
 use crate::{Error, MResult, RankedMap};
@@ -13,21 +14,37 @@ use crate::{Error, MResult, RankedMap};
 pub struct DocumentsAddition<D> {
     updates_store: store::Updates,
     updates_results_store: store::UpdatesResults,
-    updates_notifier: crossbeam_channel::Sender<()>,
+    updates_notifier: UpdateEventsEmitter,
     documents: Vec<D>,
+    is_partial: bool,
 }
 
 impl<D> DocumentsAddition<D> {
     pub fn new(
         updates_store: store::Updates,
         updates_results_store: store::UpdatesResults,
-        updates_notifier: crossbeam_channel::Sender<()>,
+        updates_notifier: UpdateEventsEmitter,
     ) -> DocumentsAddition<D> {
         DocumentsAddition {
             updates_store,
             updates_results_store,
             updates_notifier,
             documents: Vec::new(),
+            is_partial: false,
+        }
+    }
+
+    pub fn new_partial(
+        updates_store: store::Updates,
+        updates_results_store: store::UpdatesResults,
+        updates_notifier: UpdateEventsEmitter,
+    ) -> DocumentsAddition<D> {
+        DocumentsAddition {
+            updates_store,
+            updates_results_store,
+            updates_notifier,
+            documents: Vec::new(),
+            is_partial: true,
         }
     }
 
@@ -39,12 +56,13 @@ impl<D> DocumentsAddition<D> {
     where
         D: serde::Serialize,
     {
-        let _ = self.updates_notifier.send(());
+        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
         let update_id = push_documents_addition(
             writer,
             self.updates_store,
             self.updates_results_store,
             self.documents,
+            self.is_partial,
         )?;
         Ok(update_id)
     }
@@ -61,6 +79,7 @@ pub fn push_documents_addition<D: serde::Serialize>(
     updates_store: store::Updates,
     updates_results_store: store::UpdatesResults,
     addition: Vec<D>,
+    is_partial: bool,
 ) -> MResult<u64> {
     let mut values = Vec::with_capacity(addition.len());
     for add in addition {
@@ -71,7 +90,12 @@ pub fn push_documents_addition<D: serde::Serialize>(
 
     let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
 
-    let update = Update::DocumentsAddition(values);
+    let update = if is_partial {
+        Update::documents_partial(values)
+    } else {
+        Update::documents_addition(values)
+    };
+
     updates_store.put_update(writer, last_update_id, &update)?;
 
     Ok(last_update_id)
@@ -84,7 +108,7 @@ pub fn apply_documents_addition<'a, 'b>(
     documents_fields_counts_store: store::DocumentsFieldsCounts,
     postings_lists_store: store::PostingsLists,
     docs_words_store: store::DocsWords,
-    addition: Vec<serde_json::Value>,
+    addition: Vec<HashMap<String, serde_json::Value>>,
 ) -> MResult<()> {
     let mut documents_additions = HashMap::new();
 
@@ -156,6 +180,102 @@ pub fn apply_documents_addition<'a, 'b>(
     )
 }
 
+pub fn apply_documents_partial_addition<'a, 'b>(
+    writer: &'a mut heed::RwTxn<'b>,
+    main_store: store::Main,
+    documents_fields_store: store::DocumentsFields,
+    documents_fields_counts_store: store::DocumentsFieldsCounts,
+    postings_lists_store: store::PostingsLists,
+    docs_words_store: store::DocsWords,
+    addition: Vec<HashMap<String, serde_json::Value>>,
+) -> MResult<()> {
+    let mut documents_additions = HashMap::new();
+
+    let schema = match main_store.schema(writer)? {
+        Some(schema) => schema,
+        None => return Err(Error::SchemaMissing),
+    };
+
+    let identifier = schema.identifier_name();
+
+    // 1. store documents ids for future deletion
+    for mut document in addition {
+        let document_id = match extract_document_id(identifier, &document)? {
+            Some(id) => id,
+            None => return Err(Error::MissingDocumentId),
+        };
+
+        let mut deserializer = Deserializer {
+            document_id,
+            reader: writer,
+            documents_fields: documents_fields_store,
+            schema: &schema,
+            attributes: None,
+        };
+
+        // retrieve the old document and
+        // update the new one with missing keys found in the old one
+        let result = Option::<HashMap<String, serde_json::Value>>::deserialize(&mut deserializer)?;
+        if let Some(old_document) = result {
+            for (key, value) in old_document {
+                document.entry(key).or_insert(value);
+            }
+        }
+
+        documents_additions.insert(document_id, document);
+    }
+
+    // 2. remove the documents posting lists
+    let number_of_inserted_documents = documents_additions.len();
+    let documents_ids = documents_additions.iter().map(|(id, _)| *id).collect();
+    apply_documents_deletion(
+        writer,
+        main_store,
+        documents_fields_store,
+        documents_fields_counts_store,
+        postings_lists_store,
+        docs_words_store,
+        documents_ids,
+    )?;
+
+    let mut ranked_map = match main_store.ranked_map(writer)? {
+        Some(ranked_map) => ranked_map,
+        None => RankedMap::default(),
+    };
+
+    let stop_words = match main_store.stop_words_fst(writer)? {
+        Some(stop_words) => stop_words,
+        None => fst::Set::default(),
+    };
+
+    // 3. index the documents fields in the stores
+    let mut indexer = RawIndexer::new(stop_words);
+
+    for (document_id, document) in documents_additions {
+        let serializer = Serializer {
+            txn: writer,
+            schema: &schema,
+            document_store: documents_fields_store,
+            document_fields_counts: documents_fields_counts_store,
+            indexer: &mut indexer,
+            ranked_map: &mut ranked_map,
+            document_id,
+        };
+
+        document.serialize(serializer)?;
+    }
+
+    write_documents_addition_index(
+        writer,
+        main_store,
+        postings_lists_store,
+        docs_words_store,
+        &ranked_map,
+        number_of_inserted_documents,
+        indexer,
+    )
+}
+
 pub fn reindex_all_documents(
     writer: &mut heed::RwTxn,
     main_store: store::Main,

meilidb-core/src/update/documents_deletion.rs

@@ -4,6 +4,7 @@ use fst::{SetBuilder, Streamer};
 use meilidb_schema::Schema;
 use sdset::{duo::DifferenceByKey, SetBuf, SetOperation};
 
+use crate::database::{UpdateEvent, UpdateEventsEmitter};
 use crate::serde::extract_document_id;
 use crate::store;
 use crate::update::{next_update_id, Update};
@@ -12,7 +13,7 @@ use crate::{DocumentId, Error, MResult, RankedMap};
 pub struct DocumentsDeletion {
     updates_store: store::Updates,
     updates_results_store: store::UpdatesResults,
-    updates_notifier: crossbeam_channel::Sender<()>,
+    updates_notifier: UpdateEventsEmitter,
     documents: Vec<DocumentId>,
 }
 
@@ -20,7 +21,7 @@ impl DocumentsDeletion {
     pub fn new(
         updates_store: store::Updates,
         updates_results_store: store::UpdatesResults,
-        updates_notifier: crossbeam_channel::Sender<()>,
+        updates_notifier: UpdateEventsEmitter,
     ) -> DocumentsDeletion {
         DocumentsDeletion {
             updates_store,
@@ -50,7 +51,7 @@ impl DocumentsDeletion {
     }
 
     pub fn finalize(self, writer: &mut heed::RwTxn) -> MResult<u64> {
-        let _ = self.updates_notifier.send(());
+        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
         let update_id = push_documents_deletion(
             writer,
             self.updates_store,
@@ -75,7 +76,7 @@ pub fn push_documents_deletion(
 ) -> MResult<u64> {
     let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
 
-    let update = Update::DocumentsDeletion(deletion);
+    let update = Update::documents_deletion(deletion);
     updates_store.put_update(writer, last_update_id, &update)?;
 
     Ok(last_update_id)

meilidb-core/src/update/mod.rs

@@ -10,7 +10,9 @@ mod synonyms_deletion;
 
 pub use self::clear_all::{apply_clear_all, push_clear_all};
 pub use self::customs_update::{apply_customs_update, push_customs_update};
-pub use self::documents_addition::{apply_documents_addition, DocumentsAddition};
+pub use self::documents_addition::{
+    apply_documents_addition, apply_documents_partial_addition, DocumentsAddition,
+};
 pub use self::documents_deletion::{apply_documents_deletion, DocumentsDeletion};
 pub use self::schema_update::{apply_schema_update, push_schema_update};
 pub use self::stop_words_addition::{apply_stop_words_addition, StopWordsAddition};
@@ -19,9 +21,10 @@ pub use self::synonyms_addition::{apply_synonyms_addition, SynonymsAddition};
 pub use self::synonyms_deletion::{apply_synonyms_deletion, SynonymsDeletion};
 
 use std::cmp;
-use std::collections::{BTreeMap, BTreeSet};
-use std::time::{Duration, Instant};
+use std::collections::{BTreeMap, BTreeSet, HashMap};
+use std::time::Instant;
 
+use chrono::{DateTime, Utc};
 use heed::Result as ZResult;
 use log::debug;
 use serde::{Deserialize, Serialize};
@@ -30,11 +33,90 @@ use crate::{store, DocumentId, MResult};
 use meilidb_schema::Schema;
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
-pub enum Update {
+pub struct Update {
+    data: UpdateData,
+    enqueued_at: DateTime<Utc>,
+}
+
+impl Update {
+    fn clear_all() -> Update {
+        Update {
+            data: UpdateData::ClearAll,
+            enqueued_at: Utc::now(),
+        }
+    }
+
+    fn schema(data: Schema) -> Update {
+        Update {
+            data: UpdateData::Schema(data),
+            enqueued_at: Utc::now(),
+        }
+    }
+
+    fn customs(data: Vec<u8>) -> Update {
+        Update {
+            data: UpdateData::Customs(data),
+            enqueued_at: Utc::now(),
+        }
+    }
+
+    fn documents_addition(data: Vec<HashMap<String, serde_json::Value>>) -> Update {
+        Update {
+            data: UpdateData::DocumentsAddition(data),
+            enqueued_at: Utc::now(),
+        }
+    }
+
+    fn documents_partial(data: Vec<HashMap<String, serde_json::Value>>) -> Update {
+        Update {
+            data: UpdateData::DocumentsPartial(data),
+            enqueued_at: Utc::now(),
+        }
+    }
+
+    fn documents_deletion(data: Vec<DocumentId>) -> Update {
+        Update {
+            data: UpdateData::DocumentsDeletion(data),
+            enqueued_at: Utc::now(),
+        }
+    }
+
+    fn synonyms_addition(data: BTreeMap<String, Vec<String>>) -> Update {
+        Update {
+            data: UpdateData::SynonymsAddition(data),
+            enqueued_at: Utc::now(),
+        }
+    }
+
+    fn synonyms_deletion(data: BTreeMap<String, Option<Vec<String>>>) -> Update {
+        Update {
+            data: UpdateData::SynonymsDeletion(data),
+            enqueued_at: Utc::now(),
+        }
+    }
+
+    fn stop_words_addition(data: BTreeSet<String>) -> Update {
+        Update {
+            data: UpdateData::StopWordsAddition(data),
+            enqueued_at: Utc::now(),
+        }
+    }
+
+    fn stop_words_deletion(data: BTreeSet<String>) -> Update {
+        Update {
+            data: UpdateData::StopWordsDeletion(data),
+            enqueued_at: Utc::now(),
+        }
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum UpdateData {
     ClearAll,
     Schema(Schema),
     Customs(Vec<u8>),
-    DocumentsAddition(Vec<serde_json::Value>),
+    DocumentsAddition(Vec<HashMap<String, serde_json::Value>>),
+    DocumentsPartial(Vec<HashMap<String, serde_json::Value>>),
     DocumentsDeletion(Vec<DocumentId>),
     SynonymsAddition(BTreeMap<String, Vec<String>>),
     SynonymsDeletion(BTreeMap<String, Option<Vec<String>>>),
@@ -42,30 +124,31 @@ pub enum Update {
     StopWordsDeletion(BTreeSet<String>),
 }
 
-impl Update {
+impl UpdateData {
     pub fn update_type(&self) -> UpdateType {
         match self {
-            Update::ClearAll => UpdateType::ClearAll,
-            Update::Schema(schema) => UpdateType::Schema {
-                schema: schema.clone(),
-            },
-            Update::Customs(_) => UpdateType::Customs,
-            Update::DocumentsAddition(addition) => UpdateType::DocumentsAddition {
+            UpdateData::ClearAll => UpdateType::ClearAll,
+            UpdateData::Schema(_) => UpdateType::Schema,
+            UpdateData::Customs(_) => UpdateType::Customs,
+            UpdateData::DocumentsAddition(addition) => UpdateType::DocumentsAddition {
                 number: addition.len(),
             },
-            Update::DocumentsDeletion(deletion) => UpdateType::DocumentsDeletion {
+            UpdateData::DocumentsPartial(addition) => UpdateType::DocumentsPartial {
+                number: addition.len(),
+            },
+            UpdateData::DocumentsDeletion(deletion) => UpdateType::DocumentsDeletion {
                 number: deletion.len(),
             },
-            Update::SynonymsAddition(addition) => UpdateType::SynonymsAddition {
+            UpdateData::SynonymsAddition(addition) => UpdateType::SynonymsAddition {
                 number: addition.len(),
             },
-            Update::SynonymsDeletion(deletion) => UpdateType::SynonymsDeletion {
+            UpdateData::SynonymsDeletion(deletion) => UpdateType::SynonymsDeletion {
                 number: deletion.len(),
             },
-            Update::StopWordsAddition(addition) => UpdateType::StopWordsAddition {
+            UpdateData::StopWordsAddition(addition) => UpdateType::StopWordsAddition {
                 number: addition.len(),
             },
-            Update::StopWordsDeletion(deletion) => UpdateType::StopWordsDeletion {
+            UpdateData::StopWordsDeletion(deletion) => UpdateType::StopWordsDeletion {
                 number: deletion.len(),
             },
         }
@@ -73,11 +156,13 @@ impl Update {
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(tag = "name")]
 pub enum UpdateType {
     ClearAll,
-    Schema { schema: Schema },
+    Schema,
     Customs,
     DocumentsAddition { number: usize },
+    DocumentsPartial { number: usize },
     DocumentsDeletion { number: usize },
     SynonymsAddition { number: usize },
     SynonymsDeletion { number: usize },
@@ -85,30 +170,36 @@ pub enum UpdateType {
     StopWordsDeletion { number: usize },
 }
 
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct DetailedDuration {
-    pub main: Duration,
-}
-
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct ProcessedUpdateResult {
     pub update_id: u64,
+    #[serde(rename = "type")]
     pub update_type: UpdateType,
-    pub result: Result<(), String>,
-    pub detailed_duration: DetailedDuration,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub error: Option<String>,
+    pub duration: f64, // in seconds
+    pub enqueued_at: DateTime<Utc>,
+    pub processed_at: DateTime<Utc>,
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct EnqueuedUpdateResult {
     pub update_id: u64,
     pub update_type: UpdateType,
+    pub enqueued_at: DateTime<Utc>,
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase", tag = "status")]
 pub enum UpdateStatus {
-    Enqueued(EnqueuedUpdateResult),
-    Processed(ProcessedUpdateResult),
-    Unknown,
+    Enqueued {
+        #[serde(flatten)]
+        content: EnqueuedUpdateResult,
+    },
+    Processed {
+        #[serde(flatten)]
+        content: ProcessedUpdateResult,
+    },
 }
 
 pub fn update_status(
@@ -116,19 +207,19 @@ pub fn update_status(
     updates_store: store::Updates,
     updates_results_store: store::UpdatesResults,
     update_id: u64,
-) -> MResult<UpdateStatus> {
+) -> MResult<Option<UpdateStatus>> {
     match updates_results_store.update_result(reader, update_id)? {
-        Some(result) => Ok(UpdateStatus::Processed(result)),
-        None => {
-            if let Some(update) = updates_store.get(reader, update_id)? {
-                Ok(UpdateStatus::Enqueued(EnqueuedUpdateResult {
+        Some(result) => Ok(Some(UpdateStatus::Processed { content: result })),
+        None => match updates_store.get(reader, update_id)? {
+            Some(update) => Ok(Some(UpdateStatus::Enqueued {
+                content: EnqueuedUpdateResult {
                     update_id,
-                    update_type: update.update_type(),
-                }))
-            } else {
-                Ok(UpdateStatus::Unknown)
-            }
-        }
+                    update_type: update.data.update_type(),
+                    enqueued_at: update.enqueued_at,
+                },
+            })),
+            None => Ok(None),
+        },
     }
 }
 
@@ -157,8 +248,10 @@ pub fn update_task<'a, 'b>(
 ) -> MResult<ProcessedUpdateResult> {
     debug!("Processing update number {}", update_id);
 
-    let (update_type, result, duration) = match update {
-        Update::ClearAll => {
+    let Update { enqueued_at, data } = update;
+
+    let (update_type, result, duration) = match data {
+        UpdateData::ClearAll => {
             let start = Instant::now();
 
             let update_type = UpdateType::ClearAll;
@@ -173,12 +266,10 @@ pub fn update_task<'a, 'b>(
 
             (update_type, result, start.elapsed())
         }
-        Update::Schema(schema) => {
+        UpdateData::Schema(schema) => {
             let start = Instant::now();
 
-            let update_type = UpdateType::Schema {
-                schema: schema.clone(),
-            };
+            let update_type = UpdateType::Schema;
             let result = apply_schema_update(
                 writer,
                 &schema,
@@ -191,7 +282,7 @@ pub fn update_task<'a, 'b>(
 
             (update_type, result, start.elapsed())
         }
-        Update::Customs(customs) => {
+        UpdateData::Customs(customs) => {
             let start = Instant::now();
 
             let update_type = UpdateType::Customs;
@@ -199,7 +290,7 @@ pub fn update_task<'a, 'b>(
 
             (update_type, result, start.elapsed())
         }
-        Update::DocumentsAddition(documents) => {
+        UpdateData::DocumentsAddition(documents) => {
             let start = Instant::now();
 
             let update_type = UpdateType::DocumentsAddition {
@@ -218,7 +309,26 @@ pub fn update_task<'a, 'b>(
 
             (update_type, result, start.elapsed())
         }
-        Update::DocumentsDeletion(documents) => {
+        UpdateData::DocumentsPartial(documents) => {
+            let start = Instant::now();
+
+            let update_type = UpdateType::DocumentsPartial {
+                number: documents.len(),
+            };
+
+            let result = apply_documents_partial_addition(
+                writer,
+                index.main,
+                index.documents_fields,
+                index.documents_fields_counts,
+                index.postings_lists,
+                index.docs_words,
+                documents,
+            );
+
+            (update_type, result, start.elapsed())
+        }
+        UpdateData::DocumentsDeletion(documents) => {
             let start = Instant::now();
 
             let update_type = UpdateType::DocumentsDeletion {
@@ -237,7 +347,7 @@ pub fn update_task<'a, 'b>(
 
             (update_type, result, start.elapsed())
         }
-        Update::SynonymsAddition(synonyms) => {
+        UpdateData::SynonymsAddition(synonyms) => {
             let start = Instant::now();
 
             let update_type = UpdateType::SynonymsAddition {
@@ -248,7 +358,7 @@ pub fn update_task<'a, 'b>(
 
             (update_type, result, start.elapsed())
         }
-        Update::SynonymsDeletion(synonyms) => {
+        UpdateData::SynonymsDeletion(synonyms) => {
             let start = Instant::now();
 
             let update_type = UpdateType::SynonymsDeletion {
@@ -259,7 +369,7 @@ pub fn update_task<'a, 'b>(
 
             (update_type, result, start.elapsed())
         }
-        Update::StopWordsAddition(stop_words) => {
+        UpdateData::StopWordsAddition(stop_words) => {
             let start = Instant::now();
 
             let update_type = UpdateType::StopWordsAddition {
@@ -271,7 +381,7 @@ pub fn update_task<'a, 'b>(
 
             (update_type, result, start.elapsed())
         }
-        Update::StopWordsDeletion(stop_words) => {
+        UpdateData::StopWordsDeletion(stop_words) => {
             let start = Instant::now();
 
             let update_type = UpdateType::StopWordsDeletion {
@@ -297,12 +407,13 @@ pub fn update_task<'a, 'b>(
         update_id, update_type, result
     );
 
-    let detailed_duration = DetailedDuration { main: duration };
     let status = ProcessedUpdateResult {
         update_id,
         update_type,
-        result: result.map_err(|e| e.to_string()),
-        detailed_duration,
+        error: result.map_err(|e| e.to_string()).err(),
+        duration: duration.as_secs_f64(),
+        enqueued_at,
+        processed_at: Utc::now(),
     };
 
     Ok(status)

meilidb-core/src/update/schema_update.rs

@@ -14,7 +14,7 @@ pub fn apply_schema_update(
     docs_words_store: store::DocsWords,
 ) -> MResult<()> {
     use UnsupportedOperation::{
-        CannotIntroduceNewSchemaAttribute, CannotRemoveSchemaAttribute,
+        CanOnlyIntroduceNewSchemaAttributesAtEnd, CannotRemoveSchemaAttribute,
         CannotReorderSchemaAttribute, CannotUpdateSchemaIdentifier,
     };
 
@@ -33,7 +33,12 @@ pub fn apply_schema_update(
                         need_full_reindexing = true;
                     }
                 }
-                Diff::NewAttr { .. } => return Err(CannotIntroduceNewSchemaAttribute.into()),
+                Diff::NewAttr { pos, .. } => {
+                    // new attribute not at the end of the schema
+                    if pos < old_schema.number_of_attributes() {
+                        return Err(CanOnlyIntroduceNewSchemaAttributesAtEnd.into());
+                    }
+                }
                 Diff::RemovedAttr { .. } => return Err(CannotRemoveSchemaAttribute.into()),
             }
         }
@@ -63,7 +68,7 @@ pub fn push_schema_update(
 ) -> MResult<u64> {
     let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
 
-    let update = Update::Schema(schema);
+    let update = Update::schema(schema);
     updates_store.put_update(writer, last_update_id, &update)?;
 
     Ok(last_update_id)

meilidb-core/src/update/stop_words_addition.rs

@@ -3,13 +3,14 @@ use std::collections::BTreeSet;
 use fst::{set::OpBuilder, SetBuilder};
 
 use crate::automaton::normalize_str;
+use crate::database::{UpdateEvent, UpdateEventsEmitter};
 use crate::update::{next_update_id, Update};
 use crate::{store, MResult};
 
 pub struct StopWordsAddition {
     updates_store: store::Updates,
     updates_results_store: store::UpdatesResults,
-    updates_notifier: crossbeam_channel::Sender<()>,
+    updates_notifier: UpdateEventsEmitter,
     stop_words: BTreeSet<String>,
 }
 
@@ -17,7 +18,7 @@ impl StopWordsAddition {
     pub fn new(
         updates_store: store::Updates,
         updates_results_store: store::UpdatesResults,
-        updates_notifier: crossbeam_channel::Sender<()>,
+        updates_notifier: UpdateEventsEmitter,
     ) -> StopWordsAddition {
         StopWordsAddition {
             updates_store,
@@ -33,7 +34,7 @@ impl StopWordsAddition {
     }
 
     pub fn finalize(self, writer: &mut heed::RwTxn) -> MResult<u64> {
-        let _ = self.updates_notifier.send(());
+        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
         let update_id = push_stop_words_addition(
             writer,
             self.updates_store,
@@ -52,7 +53,7 @@ pub fn push_stop_words_addition(
 ) -> MResult<u64> {
     let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
 
-    let update = Update::StopWordsAddition(addition);
+    let update = Update::stop_words_addition(addition);
     updates_store.put_update(writer, last_update_id, &update)?;
 
     Ok(last_update_id)

meilidb-core/src/update/stop_words_deletion.rs

@@ -3,6 +3,7 @@ use std::collections::BTreeSet;
 use fst::{set::OpBuilder, SetBuilder};
 
 use crate::automaton::normalize_str;
+use crate::database::{UpdateEvent, UpdateEventsEmitter};
 use crate::update::documents_addition::reindex_all_documents;
 use crate::update::{next_update_id, Update};
 use crate::{store, MResult};
@@ -10,7 +11,7 @@ use crate::{store, MResult};
 pub struct StopWordsDeletion {
     updates_store: store::Updates,
     updates_results_store: store::UpdatesResults,
-    updates_notifier: crossbeam_channel::Sender<()>,
+    updates_notifier: UpdateEventsEmitter,
     stop_words: BTreeSet<String>,
 }
 
@@ -18,7 +19,7 @@ impl StopWordsDeletion {
     pub fn new(
         updates_store: store::Updates,
         updates_results_store: store::UpdatesResults,
-        updates_notifier: crossbeam_channel::Sender<()>,
+        updates_notifier: UpdateEventsEmitter,
     ) -> StopWordsDeletion {
         StopWordsDeletion {
             updates_store,
@@ -34,7 +35,7 @@ impl StopWordsDeletion {
     }
 
     pub fn finalize(self, writer: &mut heed::RwTxn) -> MResult<u64> {
-        let _ = self.updates_notifier.send(());
+        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
         let update_id = push_stop_words_deletion(
             writer,
             self.updates_store,
@@ -53,7 +54,7 @@ pub fn push_stop_words_deletion(
 ) -> MResult<u64> {
     let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
 
-    let update = Update::StopWordsDeletion(deletion);
+    let update = Update::stop_words_deletion(deletion);
     updates_store.put_update(writer, last_update_id, &update)?;
 
     Ok(last_update_id)

meilidb-core/src/update/synonyms_addition.rs

@@ -4,13 +4,14 @@ use fst::{set::OpBuilder, SetBuilder};
 use sdset::SetBuf;
 
 use crate::automaton::normalize_str;
+use crate::database::{UpdateEvent, UpdateEventsEmitter};
 use crate::update::{next_update_id, Update};
 use crate::{store, MResult};
 
 pub struct SynonymsAddition {
     updates_store: store::Updates,
     updates_results_store: store::UpdatesResults,
-    updates_notifier: crossbeam_channel::Sender<()>,
+    updates_notifier: UpdateEventsEmitter,
     synonyms: BTreeMap<String, Vec<String>>,
 }
 
@@ -18,7 +19,7 @@ impl SynonymsAddition {
     pub fn new(
         updates_store: store::Updates,
         updates_results_store: store::UpdatesResults,
-        updates_notifier: crossbeam_channel::Sender<()>,
+        updates_notifier: UpdateEventsEmitter,
     ) -> SynonymsAddition {
         SynonymsAddition {
             updates_store,
@@ -43,7 +44,7 @@ impl SynonymsAddition {
     }
 
     pub fn finalize(self, writer: &mut heed::RwTxn) -> MResult<u64> {
-        let _ = self.updates_notifier.send(());
+        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
         let update_id = push_synonyms_addition(
             writer,
             self.updates_store,
@@ -62,7 +63,7 @@ pub fn push_synonyms_addition(
 ) -> MResult<u64> {
     let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
 
-    let update = Update::SynonymsAddition(addition);
+    let update = Update::synonyms_addition(addition);
     updates_store.put_update(writer, last_update_id, &update)?;
 
     Ok(last_update_id)

meilidb-core/src/update/synonyms_deletion.rs

@@ -5,13 +5,14 @@ use fst::{set::OpBuilder, SetBuilder};
 use sdset::SetBuf;
 
 use crate::automaton::normalize_str;
+use crate::database::{UpdateEvent, UpdateEventsEmitter};
 use crate::update::{next_update_id, Update};
 use crate::{store, MResult};
 
 pub struct SynonymsDeletion {
     updates_store: store::Updates,
     updates_results_store: store::UpdatesResults,
-    updates_notifier: crossbeam_channel::Sender<()>,
+    updates_notifier: UpdateEventsEmitter,
     synonyms: BTreeMap<String, Option<Vec<String>>>,
 }
 
@@ -19,7 +20,7 @@ impl SynonymsDeletion {
     pub fn new(
         updates_store: store::Updates,
         updates_results_store: store::UpdatesResults,
-        updates_notifier: crossbeam_channel::Sender<()>,
+        updates_notifier: UpdateEventsEmitter,
     ) -> SynonymsDeletion {
         SynonymsDeletion {
             updates_store,
@@ -50,7 +51,7 @@ impl SynonymsDeletion {
     }
 
     pub fn finalize(self, writer: &mut heed::RwTxn) -> MResult<u64> {
-        let _ = self.updates_notifier.send(());
+        let _ = self.updates_notifier.send(UpdateEvent::NewUpdate);
         let update_id = push_synonyms_deletion(
             writer,
             self.updates_store,
@@ -69,7 +70,7 @@ pub fn push_synonyms_deletion(
 ) -> MResult<u64> {
     let last_update_id = next_update_id(writer, updates_store, updates_results_store)?;
 
-    let update = Update::SynonymsDeletion(deletion);
+    let update = Update::synonyms_deletion(deletion);
     updates_store.put_update(writer, last_update_id, &update)?;
 
     Ok(last_update_id)

meilidb-http/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "meilidb-http"
-version = "0.1.0"
+version = "0.2.0"
 authors = [
     "Quentin de Quelen <quentin@dequelen.me>",
     "Clément Renault <clement@meilisearch.com>",
@@ -10,7 +10,8 @@ edition = "2018"
 [dependencies]
 bincode = "1.2.0"
 chrono = { version = "0.4.9", features = ["serde"] }
-crossbeam-channel = "0.3.9"
+crossbeam-channel = "0.4.0"
+env_logger = "0.7.1"
 envconfig = "0.5.1"
 envconfig_derive = "0.5.1"
 heed = "0.5.0"
@@ -19,7 +20,7 @@ indexmap = { version = "1.3.0", features = ["serde-1"] }
 jemallocator = "0.3.2"
 log = "0.4.8"
 main_error = "0.1.0"
-meilidb-core = { path = "../meilidb-core", version = "0.6.0" }
+meilidb-core = { path = "../meilidb-core", version = "0.7.0" }
 meilidb-schema = { path = "../meilidb-schema", version = "0.6.0" }
 pretty-bytes = "0.2.2"
 rand = "0.7.2"
@@ -33,7 +34,7 @@ walkdir = "2.2.9"
 [dependencies.async-compression]
 default-features = false
 features = ["stream", "gzip", "zlib", "brotli", "zstd"]
-version = "0.1.0-alpha.7"
+version = "=0.1.0-alpha.7"
 
 [dependencies.tide]
 git = "https://github.com/rustasync/tide"

meilidb-http/src/data.rs

@@ -1,6 +1,5 @@
 use std::collections::HashMap;
 use std::ops::Deref;
-use std::sync::atomic::{AtomicBool, Ordering};
 use std::sync::Arc;
 
 use chrono::{DateTime, Utc};
@@ -35,7 +34,6 @@ pub struct DataInner {
     pub db_path: String,
     pub admin_token: Option<String>,
     pub server_pid: Pid,
-    pub accept_updates: Arc<AtomicBool>,
 }
 
 impl DataInner {
@@ -70,25 +68,6 @@ impl DataInner {
             .map_err(Into::into)
     }
 
-    pub fn last_backup(&self, reader: &heed::RoTxn) -> MResult<Option<DateTime<Utc>>> {
-        match self
-            .db
-            .common_store()
-            .get::<Str, SerdeDatetime>(&reader, "last-backup")?
-        {
-            Some(datetime) => Ok(Some(datetime)),
-            None => Ok(None),
-        }
-    }
-
-    pub fn set_last_backup(&self, writer: &mut heed::RwTxn) -> MResult<()> {
-        self.db
-            .common_store()
-            .put::<Str, SerdeDatetime>(writer, "last-backup", &Utc::now())?;
-
-        Ok(())
-    }
-
     pub fn fields_frequency(
         &self,
         reader: &heed::RoTxn,
@@ -143,14 +122,6 @@ impl DataInner {
 
         Ok(())
     }
-
-    pub fn stop_accept_updates(&self) {
-        self.accept_updates.store(false, Ordering::Relaxed);
-    }
-
-    pub fn accept_updates(&self) -> bool {
-        self.accept_updates.load(Ordering::Relaxed)
-    }
 }
 
 impl Data {
@@ -160,30 +131,22 @@ impl Data {
         let server_pid = sysinfo::get_current_pid().unwrap();
 
         let db = Arc::new(Database::open_or_create(opt.database_path.clone()).unwrap());
-        let accept_updates = Arc::new(AtomicBool::new(true));
 
         let inner_data = DataInner {
             db: db.clone(),
             db_path,
             admin_token,
             server_pid,
-            accept_updates,
         };
 
         let data = Data {
             inner: Arc::new(inner_data),
         };
 
-        for index_name in db.indexes_names().unwrap() {
-            let callback_context = data.clone();
-            let callback_name = index_name.clone();
-            db.set_update_callback(
-                index_name,
-                Box::new(move |status| {
-                    index_update_callback(&callback_name, &callback_context, status);
-                }),
-            );
-        }
+        let callback_context = data.clone();
+        db.set_update_callback(Box::new(move |index_name, status| {
+            index_update_callback(&index_name, &callback_context, status);
+        }));
 
         data
     }

meilidb-http/src/error.rs

@@ -17,6 +17,7 @@ pub enum ResponseError {
     DocumentNotFound(String),
     MissingHeader(String),
     BadParameter(String, String),
+    OpenIndex(String),
     CreateIndex(String),
     Maintenance,
 }
@@ -54,6 +55,10 @@ impl ResponseError {
         ResponseError::BadParameter(name.to_string(), message.to_string())
     }
 
+    pub fn open_index(message: impl Display) -> ResponseError {
+        ResponseError::OpenIndex(message.to_string())
+    }
+
     pub fn create_index(message: impl Display) -> ResponseError {
         ResponseError::CreateIndex(message.to_string())
     }
@@ -96,6 +101,10 @@ impl IntoResponse for ResponseError {
                 format!("Impossible to create index; {}", err),
                 StatusCode::BAD_REQUEST,
             ),
+            ResponseError::OpenIndex(err) => error(
+                format!("Impossible to open index; {}", err),
+                StatusCode::BAD_REQUEST,
+            ),
             ResponseError::Maintenance => error(
                 String::from("Server is in maintenance, please try again later"),
                 StatusCode::SERVICE_UNAVAILABLE,

meilidb-http/src/helpers/meilidb.rs

@@ -131,6 +131,12 @@ impl<'a> SearchBuilder<'a> {
         self
     }
 
+    pub fn add_attribute_to_search_in(&mut self, value: String) -> &SearchBuilder {
+        let attributes_to_search_in = self.attributes_to_search_in.get_or_insert(HashSet::new());
+        attributes_to_search_in.insert(value);
+        self
+    }
+
     pub fn attributes_to_highlight(&mut self, value: HashSet<String>) -> &SearchBuilder {
         self.attributes_to_highlight = Some(value);
         self
@@ -229,43 +235,35 @@ impl<'a> SearchBuilder<'a> {
                 }
                 fields = Some(set);
             }
-            let mut document: IndexMap<String, Value> = self
+
+            let document: IndexMap<String, Value> = self
                 .index
                 .document(reader, fields.as_ref(), doc.id)
                 .map_err(|e| Error::RetrieveDocument(doc.id.0, e.to_string()))?
                 .ok_or(Error::DocumentNotFound(doc.id.0))?;
 
+            let mut formatted = document.clone();
             let mut matches = doc.highlights.clone();
 
             // Crops fields if needed
-            if let Some(fields) = self.attributes_to_crop.clone() {
-                for (field, length) in fields {
-                    let _ = crop_document(&mut document, &mut matches, &schema, &field, length);
-                }
+            if let Some(fields) = &self.attributes_to_crop {
+                crop_document(&mut formatted, &mut matches, &schema, fields);
             }
 
             // Transform to readable matches
             let matches = calculate_matches(matches, self.attributes_to_retrieve.clone(), &schema);
 
             if !self.matches {
-                if let Some(attributes_to_highlight) = self.attributes_to_highlight.clone() {
-                    let highlights = calculate_highlights(
-                        document.clone(),
-                        matches.clone(),
-                        attributes_to_highlight,
-                    );
-                    for (key, value) in highlights {
-                        if let Some(content) = document.get_mut(&key) {
-                            *content = value;
-                        }
-                    }
+                if let Some(attributes_to_highlight) = &self.attributes_to_highlight {
+                    formatted = calculate_highlights(&formatted, &matches, attributes_to_highlight);
                 }
             }
 
             let matches_info = if self.matches { Some(matches) } else { None };
 
             let hit = SearchHit {
-                hit: document,
+                document,
+                formatted,
                 matches_info,
             };
 
@@ -382,7 +380,9 @@ pub type MatchesInfos = HashMap<String, Vec<MatchPosition>>;
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct SearchHit {
     #[serde(flatten)]
-    pub hit: IndexMap<String, Value>,
+    pub document: IndexMap<String, Value>,
+    #[serde(rename = "_formatted", skip_serializing_if = "IndexMap::is_empty")]
+    pub formatted: IndexMap<String, Value>,
     #[serde(rename = "_matchesInfo", skip_serializing_if = "Option::is_none")]
     pub matches_info: Option<MatchesInfos>,
 }
@@ -425,32 +425,31 @@ fn crop_document(
     document: &mut IndexMap<String, Value>,
     matches: &mut Vec<Highlight>,
     schema: &Schema,
-    field: &str,
-    length: usize,
-) -> Result<(), Error> {
+    fields: &HashMap<String, usize>,
+) {
     matches.sort_unstable_by_key(|m| (m.char_index, m.char_length));
 
-    let attribute = schema
-        .attribute(field)
-        .ok_or(Error::AttributeNotFoundOnSchema(field.to_string()))?;
-    let selected_matches = matches
-        .iter()
-        .filter(|m| SchemaAttr::new(m.attribute) == attribute)
-        .cloned();
-    let original_text = match document.get(field) {
-        Some(Value::String(text)) => text,
-        Some(_) => return Err(Error::CropFieldWrongType(field.to_string())),
-        None => return Err(Error::AttributeNotFoundOnDocument(field.to_string())),
-    };
-    let (cropped_text, cropped_matches) = crop_text(&original_text, selected_matches, length);
+    for (field, length) in fields {
+        let attribute = match schema.attribute(field) {
+            Some(attribute) => attribute,
+            None => continue,
+        };
 
-    document.insert(
-        field.to_string(),
-        serde_json::value::Value::String(cropped_text),
-    );
-    matches.retain(|m| SchemaAttr::new(m.attribute) != attribute);
-    matches.extend_from_slice(&cropped_matches);
-    Ok(())
+        let selected_matches = matches
+            .iter()
+            .filter(|m| SchemaAttr::new(m.attribute) == attribute)
+            .cloned();
+
+        if let Some(Value::String(ref mut original_text)) = document.get_mut(field) {
+            let (cropped_text, cropped_matches) =
+                crop_text(original_text, selected_matches, *length);
+
+            *original_text = cropped_text;
+
+            matches.retain(|m| SchemaAttr::new(m.attribute) != attribute);
+            matches.extend_from_slice(&cropped_matches);
+        }
+    }
 }
 
 fn calculate_matches(
@@ -490,13 +489,14 @@ fn calculate_matches(
 }
 
 fn calculate_highlights(
-    document: IndexMap<String, Value>,
-    matches: MatchesInfos,
-    attributes_to_highlight: HashSet<String>,
-) -> HighlightInfos {
-    let mut highlight_result: HashMap<String, Value> = HashMap::new();
+    document: &IndexMap<String, Value>,
+    matches: &MatchesInfos,
+    attributes_to_highlight: &HashSet<String>,
+) -> IndexMap<String, Value> {
+    let mut highlight_result = IndexMap::new();
+
     for (attribute, matches) in matches.iter() {
-        if attributes_to_highlight.contains("*") || attributes_to_highlight.contains(attribute) {
+        if attributes_to_highlight.contains(attribute) {
             if let Some(Value::String(value)) = document.get(attribute) {
                 let value: Vec<_> = value.chars().collect();
                 let mut highlighted_value = String::new();
@@ -521,6 +521,7 @@ fn calculate_highlights(
             };
         }
     }
+
     highlight_result
 }
 
@@ -537,9 +538,10 @@ mod tests {
 
         let document: IndexMap<String, Value> = serde_json::from_str(data).unwrap();
         let mut attributes_to_highlight = HashSet::new();
-        attributes_to_highlight.insert("*".to_string());
+        attributes_to_highlight.insert("title".to_string());
+        attributes_to_highlight.insert("description".to_string());
 
-        let mut matches: HashMap<String, Vec<MatchPosition>> = HashMap::new();
+        let mut matches = HashMap::new();
 
         let mut m = Vec::new();
         m.push(MatchPosition {
@@ -554,9 +556,9 @@ mod tests {
             length: 9,
         });
         matches.insert("description".to_string(), m);
-        let result = super::calculate_highlights(document, matches, attributes_to_highlight);
+        let result = super::calculate_highlights(&document, &matches, &attributes_to_highlight);
 
-        let mut result_expected = HashMap::new();
+        let mut result_expected = IndexMap::new();
         result_expected.insert(
             "title".to_string(),
             Value::String("<em>Fondation</em> (Isaac ASIMOV)".to_string()),

meilidb-http/src/lib.rs

@@ -8,4 +8,4 @@ pub mod models;
 pub mod option;
 pub mod routes;
 
-use self::data::Data;
+pub use self::data::Data;

meilidb-http/src/main.rs

@@ -7,15 +7,23 @@ use tide_log::RequestLogger;
 use meilidb_http::data::Data;
 use meilidb_http::option::Opt;
 use meilidb_http::routes;
+use meilidb_http::routes::index::index_update_callback;
 
 #[cfg(not(target_os = "macos"))]
 #[global_allocator]
 static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;
 
 pub fn main() -> Result<(), MainError> {
-    let opt = Opt::new();
+    env_logger::init();
 
+    let opt = Opt::new();
     let data = Data::new(opt.clone());
+
+    let data_cloned = data.clone();
+    data.db.set_update_callback(Box::new(move |name, status| {
+        index_update_callback(name, &data_cloned, status);
+    }));
+
     let mut app = tide::App::with_state(data);
 
     app.middleware(

meilidb-http/src/routes/document.rs

@@ -45,10 +45,6 @@ pub struct IndexUpdateResponse {
 pub async fn delete_document(ctx: Context<Data>) -> SResult<Response> {
     ctx.is_allowed(DocumentsWrite)?;
 
-    if !ctx.state().accept_updates() {
-        return Err(ResponseError::Maintenance);
-    }
-
     let index = ctx.index()?;
     let identifier = ctx.identifier()?;
     let document_id = meilidb_core::serde::compute_document_id(identifier.clone());
@@ -151,12 +147,9 @@ fn infered_schema(document: &IndexMap<String, Value>) -> Option<meilidb_schema::
     }
 }
 
-pub async fn add_or_update_multiple_documents(mut ctx: Context<Data>) -> SResult<Response> {
+async fn update_multiple_documents(mut ctx: Context<Data>, is_partial: bool) -> SResult<Response> {
     ctx.is_allowed(DocumentsWrite)?;
 
-    if !ctx.state().accept_updates() {
-        return Err(ResponseError::Maintenance);
-    }
     let data: Vec<IndexMap<String, Value>> =
         ctx.body_json().await.map_err(ResponseError::bad_request)?;
     let index = ctx.index()?;
@@ -179,7 +172,11 @@ pub async fn add_or_update_multiple_documents(mut ctx: Context<Data>) -> SResult
         }
     }
 
-    let mut document_addition = index.documents_addition();
+    let mut document_addition = if is_partial {
+        index.documents_partial_addition()
+    } else {
+        index.documents_addition()
+    };
 
     for document in data {
         document_addition.update_document(document);
@@ -197,11 +194,17 @@ pub async fn add_or_update_multiple_documents(mut ctx: Context<Data>) -> SResult
         .into_response())
 }
 
+pub async fn add_or_replace_multiple_documents(ctx: Context<Data>) -> SResult<Response> {
+    update_multiple_documents(ctx, false).await
+}
+
+pub async fn add_or_update_multiple_documents(ctx: Context<Data>) -> SResult<Response> {
+    update_multiple_documents(ctx, true).await
+}
+
 pub async fn delete_multiple_documents(mut ctx: Context<Data>) -> SResult<Response> {
     ctx.is_allowed(DocumentsWrite)?;
-    if !ctx.state().accept_updates() {
-        return Err(ResponseError::Maintenance);
-    }
+
     let data: Vec<Value> = ctx.body_json().await.map_err(ResponseError::bad_request)?;
     let index = ctx.index()?;
 
@@ -231,9 +234,7 @@ pub async fn delete_multiple_documents(mut ctx: Context<Data>) -> SResult<Respon
 
 pub async fn clear_all_documents(ctx: Context<Data>) -> SResult<Response> {
     ctx.is_allowed(DocumentsWrite)?;
-    if !ctx.state().accept_updates() {
-        return Err(ResponseError::Maintenance);
-    }
+
     let index = ctx.index()?;
 
     let env = &ctx.state().db.env;

meilidb-http/src/routes/index.rs

@@ -1,5 +1,5 @@
 use http::StatusCode;
-use meilidb_core::{ProcessedUpdateResult, UpdateStatus};
+use meilidb_core::ProcessedUpdateResult;
 use meilidb_schema::Schema;
 use serde_json::json;
 use tide::response::IntoResponse;
@@ -66,21 +66,9 @@ pub async fn create_index(mut ctx: Context<Data>) -> SResult<Response> {
 
     let created_index = match db.create_index(&index_name) {
         Ok(index) => index,
-        Err(meilidb_core::Error::IndexAlreadyExists) => db.open_index(&index_name).ok_or(
-            ResponseError::internal("index not found but must have been found"),
-        )?,
         Err(e) => return Err(ResponseError::create_index(e)),
     };
 
-    let callback_context = ctx.state().clone();
-    let callback_name = index_name.clone();
-    db.set_update_callback(
-        &index_name,
-        Box::new(move |status| {
-            index_update_callback(&callback_name, &callback_context, status);
-        }),
-    );
-
     let env = &db.env;
     let mut writer = env.write_txn().map_err(ResponseError::internal)?;
 
@@ -150,17 +138,10 @@ pub async fn get_update_status(ctx: Context<Data>) -> SResult<Response> {
         .map_err(ResponseError::internal)?;
 
     let response = match status {
-        UpdateStatus::Enqueued(data) => {
-            tide::response::json(json!({ "status": "enqueued", "data": data }))
-                .with_status(StatusCode::OK)
-                .into_response()
-        }
-        UpdateStatus::Processed(data) => {
-            tide::response::json(json!({ "status": "processed", "data": data }))
-                .with_status(StatusCode::OK)
-                .into_response()
-        }
-        UpdateStatus::Unknown => tide::response::json(json!({ "message": "unknown update id" }))
+        Some(status) => tide::response::json(status)
+            .with_status(StatusCode::OK)
+            .into_response(),
+        None => tide::response::json(json!({ "message": "unknown update id" }))
             .with_status(StatusCode::NOT_FOUND)
             .into_response(),
     };
@@ -188,15 +169,19 @@ pub async fn get_all_updates_status(ctx: Context<Data>) -> SResult<Response> {
 
 pub async fn delete_index(ctx: Context<Data>) -> SResult<StatusCode> {
     ctx.is_allowed(IndexesWrite)?;
-    let _index_name = ctx.url_param("index")?;
-    let _index = ctx.index()?;
+    let index_name = ctx.url_param("index")?;
 
-    // ctx.state()
-    //     .db
-    //     .delete_index(&index_name)
-    //     .map_err(ResponseError::internal)?;
+    let found = ctx
+        .state()
+        .db
+        .delete_index(&index_name)
+        .map_err(ResponseError::internal)?;
 
-    Ok(StatusCode::NOT_IMPLEMENTED)
+    if found {
+        Ok(StatusCode::NO_CONTENT)
+    } else {
+        Ok(StatusCode::NOT_FOUND)
+    }
 }
 
 pub fn index_update_callback(index_name: &str, data: &Data, _status: ProcessedUpdateResult) {

meilidb-http/src/routes/mod.rs

@@ -37,7 +37,8 @@ pub fn load_routes(app: &mut tide::App<Data>) {
                     router
                         .at("/")
                         .get(document::browse_documents)
-                        .post(document::add_or_update_multiple_documents)
+                        .post(document::add_or_replace_multiple_documents)
+                        .put(document::add_or_update_multiple_documents)
                         .delete(document::clear_all_documents);
 
                     router.at("/:identifier").nest(|router| {

meilidb-http/src/routes/search.rs

@@ -36,6 +36,12 @@ pub async fn search_with_url_query(ctx: Context<Data>) -> SResult<Response> {
     let env = &ctx.state().db.env;
     let reader = env.read_txn().map_err(ResponseError::internal)?;
 
+    let schema = index
+        .main
+        .schema(&reader)
+        .map_err(ResponseError::internal)?
+        .ok_or(ResponseError::open_index("No Schema found"))?;
+
     let query: SearchQuery = ctx
         .url_query()
         .map_err(|_| ResponseError::bad_request("invalid query parameter"))?;
@@ -56,23 +62,36 @@ pub async fn search_with_url_query(ctx: Context<Data>) -> SResult<Response> {
     }
     if let Some(attributes_to_search_in) = query.attributes_to_search_in {
         for attr in attributes_to_search_in.split(',') {
-            search_builder.add_retrievable_field(attr.to_string());
+            search_builder.add_attribute_to_search_in(attr.to_string());
         }
     }
     if let Some(attributes_to_crop) = query.attributes_to_crop {
         let crop_length = query.crop_length.unwrap_or(200);
-        let attributes_to_crop = attributes_to_crop
-            .split(',')
-            .map(|r| (r.to_string(), crop_length))
-            .collect();
-        search_builder.attributes_to_crop(attributes_to_crop);
+        if attributes_to_crop == "*" {
+            let attributes_to_crop = schema
+                .iter()
+                .map(|(attr, ..)| (attr.to_string(), crop_length))
+                .collect();
+            search_builder.attributes_to_crop(attributes_to_crop);
+        } else {
+            let attributes_to_crop = attributes_to_crop
+                .split(',')
+                .map(|r| (r.to_string(), crop_length))
+                .collect();
+            search_builder.attributes_to_crop(attributes_to_crop);
+        }
     }
 
     if let Some(attributes_to_highlight) = query.attributes_to_highlight {
-        let attributes_to_highlight = attributes_to_highlight
-            .split(',')
-            .map(ToString::to_string)
-            .collect();
+        let attributes_to_highlight = if attributes_to_highlight == "*" {
+            schema.iter().map(|(attr, ..)| attr.to_string()).collect()
+        } else {
+            attributes_to_highlight
+                .split(',')
+                .map(ToString::to_string)
+                .collect()
+        };
+
         search_builder.attributes_to_highlight(attributes_to_highlight);
     }
 

meilidb-schema/src/lib.rs

@@ -22,7 +22,7 @@ pub const RANKED: SchemaProps = SchemaProps {
     ranked: true,
 };
 
-#[derive(Debug, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[derive(Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
 pub struct SchemaProps {
     #[serde(default)]
     pub displayed: bool,
@@ -60,6 +60,36 @@ impl BitOr for SchemaProps {
     }
 }
 
+impl fmt::Debug for SchemaProps {
+    #[allow(non_camel_case_types)]
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        #[derive(Debug)]
+        struct DISPLAYED;
+
+        #[derive(Debug)]
+        struct INDEXED;
+
+        #[derive(Debug)]
+        struct RANKED;
+
+        let mut debug_set = f.debug_set();
+
+        if self.displayed {
+            debug_set.entry(&DISPLAYED);
+        }
+
+        if self.indexed {
+            debug_set.entry(&INDEXED);
+        }
+
+        if self.ranked {
+            debug_set.entry(&RANKED);
+        }
+
+        debug_set.finish()
+    }
+}
+
 #[derive(Serialize, Deserialize)]
 pub struct SchemaBuilder {
     identifier: String,
@@ -102,12 +132,12 @@ impl SchemaBuilder {
     }
 }
 
-#[derive(Debug, Clone, PartialEq, Eq)]
+#[derive(Clone, PartialEq, Eq)]
 pub struct Schema {
     inner: Arc<InnerSchema>,
 }
 
-#[derive(Debug, Clone, PartialEq, Eq)]
+#[derive(Clone, PartialEq, Eq)]
 struct InnerSchema {
     identifier: String,
     attrs: HashMap<String, SchemaAttr>,
@@ -139,6 +169,10 @@ impl Schema {
         attributes
     }
 
+    pub fn number_of_attributes(&self) -> usize {
+        self.inner.attrs.len()
+    }
+
     pub fn props(&self, attr: SchemaAttr) -> SchemaProps {
         let (_, props) = self.inner.props[attr.0 as usize];
         props
@@ -184,6 +218,16 @@ impl<'de> Deserialize<'de> for Schema {
     }
 }
 
+impl fmt::Debug for Schema {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        let builder = self.to_builder();
+        f.debug_struct("Schema")
+            .field("identifier", &builder.identifier)
+            .field("attributes", &builder.attributes)
+            .finish()
+    }
+}
+
 #[derive(Serialize, Deserialize, Debug, Copy, Clone, PartialOrd, Ord, PartialEq, Eq, Hash)]
 pub struct SchemaAttr(pub u16);
 
@@ -447,4 +491,43 @@ mod tests {
 
         Ok(())
     }
+
+    #[test]
+    fn debug_output() {
+        use std::fmt::Write as _;
+
+        let mut builder = SchemaBuilder::with_identifier("id");
+        builder.new_attribute("alpha", DISPLAYED);
+        builder.new_attribute("beta", DISPLAYED | INDEXED);
+        builder.new_attribute("gamma", INDEXED);
+        let schema = builder.build();
+
+        let mut output = String::new();
+        let _ = write!(&mut output, "{:#?}", schema);
+
+        let expected = r#"Schema {
+    identifier: "id",
+    attributes: {
+        "alpha": {
+            DISPLAYED,
+        },
+        "beta": {
+            DISPLAYED,
+            INDEXED,
+        },
+        "gamma": {
+            INDEXED,
+        },
+    },
+}"#;
+
+        assert_eq!(output, expected);
+
+        let mut output = String::new();
+        let _ = write!(&mut output, "{:?}", schema);
+
+        let expected = r#"Schema { identifier: "id", attributes: {"alpha": {DISPLAYED}, "beta": {DISPLAYED, INDEXED}, "gamma": {INDEXED}} }"#;
+
+        assert_eq!(output, expected);
+    }
 }

meilidb-tokenizer/Cargo.toml

@@ -1,8 +1,9 @@
 [package]
 name = "meilidb-tokenizer"
-version = "0.6.0"
+version = "0.6.1"
 authors = ["Kerollmops <renault.cle@gmail.com>"]
 edition = "2018"
 
 [dependencies]
+deunicode = "1.0.0"
 slice-group-by = "0.2.4"

meilidb-tokenizer/src/lib.rs

@@ -1,4 +1,5 @@
 use self::SeparatorCategory::*;
+use deunicode::deunicode_char;
 use slice_group_by::StrGroupBy;
 use std::iter::Peekable;
 
@@ -43,7 +44,10 @@ fn is_separator(c: char) -> bool {
 
 fn classify_separator(c: char) -> Option<SeparatorCategory> {
     match c {
-        ' ' | '-' | '_' | '\'' | ':' | '"' => Some(Soft),
+        c if c.is_whitespace() => Some(Soft), // whitespaces
+        c if deunicode_char(c) == Some("'") => Some(Soft), // quotes
+        c if deunicode_char(c) == Some("\"") => Some(Soft), // double quotes
+        '-' | '_' | '\'' | ':' | '/' | '\\' => Some(Soft),
         '.' | ';' | ',' | '!' | '?' | '(' | ')' => Some(Hard),
         _ => None,
     }

meilidb-types/Cargo.toml

@@ -0,0 +1,17 @@
+[package]
+name = "meilidb-types"
+version = "0.1.0"
+authors = ["Clément Renault <renault.cle@gmail.com>"]
+edition = "2018"
+
+[dependencies.zerocopy]
+version = "0.2.8"
+optional = true
+
+[dependencies.serde]
+version = "1.0.101"
+features = ["derive"]
+optional = true
+
+[features]
+default = ["serde", "zerocopy"]

meilidb-types/src/lib.rs

@@ -0,0 +1,65 @@
+#[cfg(feature = "zerocopy")]
+use zerocopy::{AsBytes, FromBytes};
+
+#[cfg(feature = "serde")]
+use serde::{Deserialize, Serialize};
+
+/// Represent an internally generated document unique identifier.
+///
+/// It is used to inform the database the document you want to deserialize.
+/// Helpful for custom ranking.
+#[derive(Debug, Copy, Clone, Eq, PartialEq, PartialOrd, Ord, Hash)]
+#[cfg_attr(feature = "zerocopy", derive(AsBytes, FromBytes))]
+#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
+#[repr(C)]
+pub struct DocumentId(pub u64);
+
+/// This structure represent the position of a word
+/// in a document and its attributes.
+///
+/// This is stored in the map, generated at index time,
+/// extracted and interpreted at search time.
+#[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
+#[cfg_attr(feature = "zerocopy", derive(AsBytes, FromBytes))]
+#[repr(C)]
+pub struct DocIndex {
+    /// The document identifier where the word was found.
+    pub document_id: DocumentId,
+
+    /// The attribute in the document where the word was found
+    /// along with the index in it.
+    pub attribute: u16,
+    pub word_index: u16,
+
+    /// The position in bytes where the word was found
+    /// along with the length of it.
+    ///
+    /// It informs on the original word area in the text indexed
+    /// without needing to run the tokenizer again.
+    pub char_index: u16,
+    pub char_length: u16,
+}
+
+/// This structure represent a matching word with informations
+/// on the location of the word in the document.
+///
+/// The order of the field is important because it defines
+/// the way these structures are ordered between themselves.
+#[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub struct Highlight {
+    /// The attribute in the document where the word was found
+    /// along with the index in it.
+    pub attribute: u16,
+
+    /// The position in bytes where the word was found.
+    ///
+    /// It informs on the original word area in the text indexed
+    /// without needing to run the tokenizer again.
+    pub char_index: u16,
+
+    /// The length in bytes of the found word.
+    ///
+    /// It informs on the original word area in the text indexed
+    /// without needing to run the tokenizer again.
+    pub char_length: u16,
+}