Commit Graph

7172 Commits

Author SHA1 Message Date
c6d107a05f makes the content-type mandatory for every routes 2021-10-06 17:15:47 +02:00
ddbcf449da Merge #1763
1763: Index tests r=MarinPostma a=MarinPostma

This pr aims to test more thorougly the usage on index in the meilisearch database, by writing unit tests.

work included:
- [x] Create index mock and stub methods
- [x] Test snapshot creation
- [x] Test Dumps
- [x] Test search

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-10-06 14:39:53 +00:00
9fa61439b1 fix clippy warning & unsafety 2021-10-06 14:51:46 +02:00
02dd1ea29d Merge pull request #1771 from ferdi05/ferdi05-patch-contributing
Update CONTRIBUTING.md
2021-10-06 14:40:38 +02:00
a38215de98 edit documentation 2021-10-06 14:35:18 +02:00
85b5260d9d simple search unit test 2021-10-06 14:20:05 +02:00
4b4ebad9a9 test dumps 2021-10-06 14:10:26 +02:00
ece4c739f4 update store tests 2021-10-06 14:10:26 +02:00
85ae34cf9f test snapshots 2021-10-06 14:10:23 +02:00
0448f0ce56 handle panic in stubs 2021-10-06 14:09:04 +02:00
4835d82a0b implement index mock 2021-10-06 14:09:01 +02:00
dde1da1c0e Merge #382
382: Refactor attribute criterion r=Kerollmops a=ManyTheFish

### Re-implement set based algorithm for attribute criterion
#### Levels
Instead of doing level iteration and digging in the interesting level, we only iterate over the lowest level.

#### crossword iteration VS minimal position iteration
Instead of crossing word position in order to iterate strictly over the position that gives the best rank in good order; we iterate word by word starting with the word that increases the rank the little as possible.
This new method is a bit less precise but way simpler.

### Simplify word-level-position database
We don't use levels anymore in the attribute criterion, and so we removed the level complexity of the database making a word-position-docids database.

### Benchmarks on search on big datasets

#### songs main VS refactor-attribute-criterion
```diff
  group                                                   search_songsmain_31c18f09               search_songsrefactor-attribute-criterion_1bd15d84
  -----                                                   -------------------------               -------------------------------------------------
- smol-songs.csv: basic filter: <=/Notstandskomitee       1.00     84.8±0.58µs        ? ?/sec     1.09     92.2±8.98µs        ? ?/sec
+ smol-songs.csv: basic filter: TO/Notstandskomitee       1.18     98.0±6.30µs        ? ?/sec     1.00     83.2±0.97µs        ? ?/sec
+ smol-songs.csv: basic with quote/"david" "bowie"        114.68    76.0±0.20ms        ? ?/sec    1.00    662.5±5.03µs        ? ?/sec
- smol-songs.csv: basic with quote/"john"                 1.00    197.4±1.06µs        ? ?/sec     1.05    208.1±1.53µs        ? ?/sec
+ smol-songs.csv: basic with quote/"michael" "jackson"    2.75      2.0±0.01ms        ? ?/sec     1.00    738.9±3.91µs        ? ?/sec
+ smol-songs.csv: basic without quote/david bowie         297.42  1499.3±0.86ms        ? ?/sec    1.00      5.0±0.02ms        ? ?/sec
+ smol-songs.csv: basic without quote/michael jackson     2.55      8.9±0.02ms        ? ?/sec     1.00      3.5±0.01ms        ? ?/sec
+ smol-songs.csv: big filter/john                         1.08    473.6±2.25µs        ? ?/sec     1.00    438.1±2.59µs        ? ?/sec
- smol-songs.csv: prefix search/a                         1.00    446.9±1.81µs        ? ?/sec     1.79    800.5±4.45µs        ? ?/sec
- smol-songs.csv: prefix search/b                         1.00    398.5±2.74µs        ? ?/sec     1.81    723.1±5.46µs        ? ?/sec
- smol-songs.csv: prefix search/i                         1.00    486.3±1.99µs        ? ?/sec     1.69    823.6±9.42µs        ? ?/sec
- smol-songs.csv: prefix search/s                         1.00    229.6±3.29µs        ? ?/sec     2.59    594.4±2.22µs        ? ?/sec
- smol-songs.csv: prefix search/x                         1.00    150.2±0.76µs        ? ?/sec     1.11    166.0±0.87µs        ? ?/sec
```

On songs, the new algorithm gives a big improvement on slow queries, and is slower on one char prefix search (fast queries <1ms).

#### wiki main VS refactor-attribute-criterion
```diff
  group                                                           search_wikimain_31c18f09               search_wikirefactor-attribute-criterion_1bd15d84
  -----                                                           ------------------------               ------------------------------------------------
- smol-wiki-articles.csv: basic with quote/"rock" "and" "roll"    1.00      3.2±0.01ms        ? ?/sec    1.15      3.7±0.01ms        ? ?/sec
- smol-wiki-articles.csv: basic without quote/film                1.00    351.5±2.47µs        ? ?/sec    1.13    396.8±1.63µs        ? ?/sec
+ smol-wiki-articles.csv: basic without quote/rock and roll       1.10      9.4±0.02ms        ? ?/sec    1.00      8.6±0.04ms        ? ?/sec
- smol-wiki-articles.csv: basic without quote/spain               1.00    446.0±3.23µs        ? ?/sec    1.11    496.6±7.75µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/c                         1.00    115.6±0.61µs        ? ?/sec    2.22    256.7±1.24µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/g                         1.00    189.7±2.03µs        ? ?/sec    1.57    297.0±1.35µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/j                         1.00    209.2±1.11µs        ? ?/sec    1.40    293.0±2.09µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/q                         1.00     79.0±0.44µs        ? ?/sec    1.10     87.2±0.69µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/t                         1.00    270.1±1.15µs        ? ?/sec    1.55    419.9±5.16µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/x                         1.00    244.9±1.33µs        ? ?/sec    1.07    260.9±1.95µs        ? ?/sec
- smol-wiki-articles.csv: words/Abraham machin                    1.00      8.1±0.03ms        ? ?/sec    1.17      9.4±0.02ms        ? ?/sec
- smol-wiki-articles.csv: words/Idaho Bellevue pizza              1.00     19.3±0.07ms        ? ?/sec    1.07     20.6±0.05ms        ? ?/sec
```
On wiki we have some regressions `+17%` and `+15%` on request `>1ms`.

Co-authored-by: many <maxime@meilisearch.com>
2021-10-06 09:19:33 +00:00
085bc6440c Apply PR comments 2021-10-06 11:12:26 +02:00
1bd15d849b Reduce candidates threshold 2021-10-05 18:52:14 +02:00
eaddee9fe2 Update CONTRIBUTING.md
typo + text improvement
all credits go to @guimachiavelli
2021-10-05 18:07:59 +02:00
ea4bd29d14 Apply PR comments 2021-10-05 17:35:07 +02:00
2190764162 Merge #1768
1768: Fix auth error r=irevoire a=MarinPostma

fix a small auth error, that set the invalid token error token to "hello". This was invilisble to the user because the invalid token is not returned.

thank you hawk-eye `@irevoire` 

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-10-05 15:16:14 +00:00
5ed75de0db Update infos crate 2021-10-05 13:56:12 +02:00
3296bb243c Simplify word level position DB into a word position DB 2021-10-05 12:15:02 +02:00
75d341d928 Re-implement set based algorithm for attribute criterion 2021-10-05 12:14:50 +02:00
3b91764587 fix auth error 2021-10-05 09:09:40 +02:00
0c3ec549f8 Merge #1764
1764: Bump Milli to improve geosearch errors r=curquiza a=irevoire

closes #1734 

`@curquiza,` your two examples still don't work because a filter must be composed of multiples operations; look at my screenshot to see what works and what doesn't.
Is this ok? 🤔 

`@gmourier,`
What do you think?

![image](https://user-images.githubusercontent.com/7032172/135846911-588f652d-16db-4d88-89fd-148640bac0f7.png)


And here is a screenshot with all the new errors that have been implemented

![image](https://user-images.githubusercontent.com/7032172/135854851-da469fef-0dd0-4ff1-b15e-89934ed8fb6f.png)

Co-authored-by: Tamo <tamo@meilisearch.com>
2021-10-04 17:30:22 +00:00
fca686e7f8 bump meilisearch 2021-10-04 13:52:37 +02:00
607e28749a Merge #1755
1755: Fix mini dashboard r=curquiza a=anirudhRowjee

This commit is a fix to issue #1750.
As a part of the changes to solve this issue, the following changes have
been made -
1. Route registration for static assets has been modified
2. the `mut` keyword on the `scope` has been removed.

Co-authored-by: Anirudh Rowjee <ani.rowjee@gmail.com>
v0.23.0rc0
2021-10-04 09:56:21 +00:00
bffab21b10 Changes
1. Removed redundant scope registration
2021-10-04 14:47:05 +05:30
d9165c7f77 Add option to use enviroment variable to increase rate limit 2021-10-03 13:07:40 +03:00
31c18f0953 Merge #381
381: Update version for the next release (v0.17.1) r=irevoire a=curquiza



Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-10-03 02:12:43 +00:00
2ef58ccce9 Fix formatting 2021-10-02 10:59:01 -07:00
4009804221 Creates non root user to run Meilisearch in Dockerfile 2021-10-02 10:43:13 -07:00
05d8a33a28 Update version for the next release (v0.17.1) 2021-10-02 16:21:31 +02:00
151f691609 Fixes #1750
This commit is a fix to issue #1750.
As a part of the changes to solve this issue, the following changes have
been made -
1. Route registration for static assets has been modified
2. the `mut` keyword on the `scope` has been removed.
2021-10-02 15:24:04 +05:30
c9092c72bf Merge #380
380: Reserved keyword error message r=Kerollmops a=irevoire

And I missed _another_ reserved keyword error message in the filter :(

Co-authored-by: Tamo <tamo@meilisearch.com>
2021-10-01 07:13:31 +00:00
81993b6a15 Merge #1747
1747: Add new error types for document additions r=curquiza a=MarinPostma

Adds the missing errors for the documents routes, as specified.

close #1691
close #1690


Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-09-30 15:16:44 +00:00
4eb3817b03 missing payload error 2021-09-30 16:58:13 +02:00
18cb514073 invalid content type error 2021-09-30 16:58:13 +02:00
ddd40d87a7 malformed payload error 2021-09-30 16:58:13 +02:00
137272b8de empty content type error 2021-09-30 16:58:13 +02:00
e400ae900d Merge #1746
1746: Do not commit transaction on failed updates r=irevoire a=Kerollmops

This PR fixes MeiliSearch that was always committing the transactions even when an update was invalid and the whole transaction should have been trashed. It was the source of a bug where an invalid update (with an invalid primary key) was creating an index with the specified primary key and should instead have failed and done nothing on the server.

Fixes #1735.

Co-authored-by: Kerollmops <clement@meilisearch.com>
2021-09-30 13:46:43 +00:00
c388dca5ec Check that invalid updates do not create an index with a primary key 2021-09-30 15:46:04 +02:00
6a691db7f8 Do not commit transaction on failed updates 2021-09-30 15:46:03 +02:00
ed783b67ca Merge #1742
1742: Create dumps v3 r=irevoire a=MarinPostma

The introduction of the obkv document format has changed the format of the updates, by removing the need for the document format of the addition (it is not necessary since update are store in the obkv format). This has caused breakage in the dumps that this pr solves by introducing a 3rd version of the dumps.

A v2 compat layer has been created that support the import of v2 dumps into meilisearch. This has permitted to move the compat code that existed elsewhere in meiliearch to be moved into the v2 module. The asc/desc patching is now only done for forward compatibility when loading a v2 dump, and the v3 write the asc/desc to the dump with the new syntax.




Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-09-30 13:17:30 +00:00
ee372a7b30 implement new dump v2 2021-09-30 14:49:13 +02:00
66f39aaa92 fix dump v3 2021-09-30 14:49:13 +02:00
03af99650d fix dumpv1 2021-09-30 14:49:13 +02:00
d9eba9d145 improve and test the sort error message 2021-09-30 14:38:27 +02:00
0ee67bb7d1 improve the reserved keyword error message for the filters 2021-09-30 14:38:27 +02:00
44a2ff07b1 Merge #1697
1697: Make exec binary for M1 mac available for download r=irevoire a=k-nasa

## Why

fix: https://github.com/meilisearch/MeiliSearch/issues/1661

Now, Do not supported getting exec file for m1 mac on using`download-latest.sh`.


## What

Download x86 binary when run `download-latest.sh` on m1 mac, because it can execute binary targeting x86.

## Proof

I verified like this.
I got executable binary on M1 mac 💡 

```sh
:) % arch
arm64

:) % ./download-latest.sh
Downloading MeiliSearch binary v0.21.1 for macos, architecture amd64...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   631  100   631    0     0   2035      0 --:--:-- --:--:-- --:--:--  2035
100 43.5M  100 43.5M    0     0  7826k      0  0:00:05  0:00:05 --:--:-- 9123k
MeiliSearch binary successfully downloaded as 'meilisearch' file.

Run it:
    $ ./meilisearch
Usage:
    $ ./meilisearch --help

:) % file ./meilisearch
./meilisearch: Mach-O 64-bit executable x86_64

:) % ./meilisearch --help # this is execuable
meilisearch-http 0.21.1
...
...
```

Co-authored-by: k-nasa <htilcs1115@gmail.com>
Co-authored-by: nasa <htilcs1115@gmail.com>
2021-09-30 11:33:48 +00:00
fb95540394 Merge #1748
1748: Add a link to join the cloud-hosted beta r=MarinPostma a=gmourier

The product team would like to add a link to communicate and invite users to fill out the form to test the closed beta of our cloud solution.

We have done the same thing on the documentation side https://github.com/meilisearch/documentation/pull/1148. 😇

Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>
2021-09-30 09:42:22 +00:00
be00fafb29 Add a link to join the cloud-hosted beta 2021-09-30 11:28:51 +02:00
0bc376a17b Merge #1738
1738: Update Dockerfile following the refactor r=MarinPostma a=curquiza



Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-09-30 08:08:19 +00:00