Commit Graph

12379 Commits

Author SHA1 Message Date
1e1821f002 Introduce the merge_two_obkv function to merge documents on update 2020-10-26 10:55:07 +01:00
60347a5483 Move the AvailableDocumentsIds iterator into the update module 2020-10-26 10:53:23 +01:00
b14cca2ad9 Introduce the UpdateBuilder type along with some update operations 2020-10-25 18:32:01 +01:00
adacc7977d Make the Index return default values when value don't exist 2020-10-25 18:30:24 +01:00
a7a4984175 Introduce the Transform type into the indexing system 2020-10-24 17:06:09 +02:00
b44b04d25b Serialize the CSV record values as JSON strings 2020-10-24 14:43:46 +02:00
656a851830 Introduce the Transform struct transforming CSVs
This allows us to:
  - transform a CSV, a JSON or a JSON lines data type into the same
    Grenad x Obkv streamable data type and creates the new FieldsIdsMap.
  - Extract all the documents user ids in advance to be able to delete
    the existing documents before re-indexing them.
  - Keep the last documents with the same user id avoiding duplicates
    in the same request.
2020-10-24 13:37:38 +02:00
8d82e37ec0 Introduce the AvailableDocumentsIds iterator 2020-10-23 12:07:01 +02:00
9dbf43d3e7 Update readme accordingly 2020-10-22 20:33:20 +02:00
772f4d6671 Remove not maintained cvs movies dataset 2020-10-22 20:33:20 +02:00
1b57218739 Merge #1040
1040: Update movie posters r=Kerollmops a=bidoubiwa

This PR resolves 3 issues: 

1. update posters URLs that changed
2. All posters point to a smaller image ( +- 20kb instead of 500kb+-) this was done by changing the width size from 1280 px to 500 px. 
3. Remove films that are not in the tmdb database

Co-authored-by: Charlotte Vermandel <charlottevermandel@gmail.com>
2020-10-22 16:38:41 +00:00
8767269b47 Update movie posters 2020-10-22 18:07:57 +02:00
baceaed582 Merge #1038
1038: Add Sandbox section to README.md r=LegendreM a=eskombro

This PR adds a link to [MeiliSearch Sandbox](https://sandbox.meilisearch.com/) in the README.md

Co-authored-by: Samuel Jimenez <sjimenezre@gmail.com>
2020-10-22 15:25:23 +00:00
62a28bc2a1 Add Sandbox section to README.md 2020-10-22 17:04:45 +02:00
f83caa6c40 Merge #1008
1008: Dump info r=Kerollmops a=LegendreM

fix #998 
fix #988 
fix #1009
fix #1010
fix #1033


Co-authored-by: many <maxime@meilisearch.com>
2020-10-22 14:23:50 +00:00
53b1483e71 fix pr comments 2020-10-22 16:12:55 +02:00
a0eafea200 fix tests 2020-10-22 15:46:20 +02:00
10dace305d snapshot at start 2020-10-22 15:46:20 +02:00
1eace79f77 change error message to be absolute 2020-10-22 15:46:20 +02:00
e6033e174d fix #1010 2020-10-22 15:46:20 +02:00
f1925b8f71 fix #1009 2020-10-22 15:46:20 +02:00
834f3cc192 rename folder to dir 2020-10-22 15:46:20 +02:00
e049aead16 improve dump status 2020-10-22 15:46:20 +02:00
2a4cd81c86 Add documentation to the Index methods 2020-10-22 15:44:12 +02:00
566a7c3039 Make the FieldsIdsMap serialization more stable by using a BTreeMap 2020-10-22 14:53:20 +02:00
9133f38138 Introduce the FieldsIdsMap type 2020-10-22 12:56:35 +02:00
0a9c9670e7 Merge #1028
1028: Clean external contributions r=Kerollmops a=LegendreM

We accepted some unperfect external PRs, this one is here to clean this:
-  clean PR #946 (remove changelog line and add forgotten newline)
- remove useless function after health route refacto #1026

Co-authored-by: many <maxime@meilisearch.com>
Co-authored-by: Many <legendre.maxime.isn@gmail.com>
2020-10-22 10:46:19 +00:00
1744dcebfe Merge branch 'master' into clean_external_contributions 2020-10-22 12:23:51 +02:00
802e925fd7 Switch to a JSON protocol for the front page 2020-10-21 18:26:29 +02:00
29712916e6 Merge #1034
1034: Remove outdated settings file r=Kerollmops a=bidoubiwa

Unnecessary settings files in the dataset folder should be removed. 

Co-authored-by: Charlotte Vermandel <charlottevermandel@gmail.com>
2020-10-21 15:42:48 +00:00
4d2783bb04 Remove outdated settings file 2020-10-21 17:12:10 +02:00
5caf523fd9 Move the Index to its own module 2020-10-21 15:55:48 +02:00
2210818114 Introduce the obkv heed codec 2020-10-21 15:51:48 +02:00
f6eecb855e Send a basic progressing status to the updates front page 2020-10-21 15:38:28 +02:00
4eeeccb9cd Change the UpdateStore to have different processed and pending meta types 2020-10-21 13:52:15 +02:00
16ab3e02a9 Change the UpdateStore internal meta serializer 2020-10-21 13:42:49 +02:00
f948a03be2 Optimise the merge functions to avoid allocations 2020-10-20 16:40:50 +02:00
50f0fbb05c remove useless function after health route refacto #1026 2020-10-20 16:21:46 +02:00
cde8478388 Replace the panic in the merge function by actual errors 2020-10-20 16:19:07 +02:00
8ed8abb9df Introduce an append-only indexing system 2020-10-20 15:00:58 +02:00
a122d3d466 Export the indexing part into a module 2020-10-20 14:22:09 +02:00
eb92e72e6c Updates can send progress update status 2020-10-20 12:28:10 +02:00
341046c96c Remove the js map file from the filesize.js script 2020-10-20 12:20:42 +02:00
3a934b7020 Split the update attributes on the updates front page 2020-10-20 12:19:48 +02:00
03ca1ff634 Make the updates page interactive 2020-10-20 12:09:38 +02:00
35c9a3c558 Brodacast the updates infos to every ws clients 2020-10-20 11:19:34 +02:00
56c3a61d83 Introduce a new updates page 2020-10-19 19:57:15 +02:00
5a842ec94a clean PR #946 2020-10-19 17:16:25 +02:00
871222aebd Introduce some new routes to handle live indexing 2020-10-19 16:06:43 +02:00
372680e2ab Merge #1026
1026: refactor /health  r=LegendreM a=frbimo

Fixes: #940 

Testing:
`cargo test` and `cargo build --release` passed

Co-authored-by: frbimo <fr.bimo@gmail.com>
2020-10-19 13:57:15 +00:00