snowykami/meilisearch
1
0
Fork 0
mirror of https://github.com/meilisearch/meilisearch.git synced 2025-11-04 09:56:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
9,704 Commits 259 Branches 544 Tags
2cacc448b6ca2080694f763b300a5d0120c30ec4
Commit Graph

9704 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Louis Dureuil
2cacc448b6 Rename src/search.rs -> src/search/mod.rs 2024-07-15 08:43:21 +02:00
Louis Dureuil
a61b852695 Add tests 2024-07-15 08:43:21 +02:00
Louis Dureuil
3167411e98 Analytics 2024-07-15 08:43:21 +02:00
Louis Dureuil
83d71662aa Changes to multi_search route 2024-07-15 08:43:21 +02:00
Louis Dureuil
5c323cecc7 search: introduce federated search 2024-07-15 08:43:21 +02:00
Louis Dureuil
d3a6d2a6fa search: introduce hitmaker 2024-07-11 16:35:59 +02:00
Louis Dureuil
2123d76089 search: introduce "search_from_kind" 2024-07-11 16:35:11 +02:00
Louis Dureuil
edab4e75b0 Make SearchKind cloneable 2024-07-11 16:33:24 +02:00
Louis Dureuil
b9982587d4 Add new errors to meilisearch 2024-07-11 16:31:44 +02:00
Louis Dureuil
e83da00446 Milli changes to match to allow for more flexible lifetimes 2024-07-11 16:29:35 +02:00
Louis Dureuil
7fb3e378ff Do not fail sort comparisons when the field name or target point are different 2024-07-11 16:28:14 +02:00
Louis Dureuil
12a7a45930 Add roaring to meilisearch 2024-07-11 16:27:50 +02:00
meili-bors[bot]
29b44e5541 Merge #4626 
4626: Edit Documents with Rhai r=ManyTheFish a=Kerollmops

This PR introduces a first version of [the _Update Documents with Function_ (internal)](https://www.notion.so/meilisearch/Update-Documents-by-Function-45f87b13e61c4435b73943768a490808). It uses [the Rhai programming language](https://rhai.rs/) to let users express the modifications they want apply.

You can read more about the way to use this functions on [the Usage PRD Page](https://meilisearch.notion.site/Edit-Documents-with-Rhai-0cff8fea7655436592e7c8a6de932062?pvs=25). The [prototype is available](https://github.com/meilisearch/meilisearch/actions/runs/9038384483) through Docker by using the following command:

```
docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-edit-documents-with-rhai-3
```

## TODO
 - [x] Support the `DocumentEdition` task in dumps.
 - [x] Remove the unwraps and panics.
 - [x] Improve error codes for the `function` parameter.
 - [x] [Update Rhai to v1.19.0](https://github.com/rhaiscript/rhai/releases/tag/v1.19.0) 🚀
 - [x] Make it an experimental feature (only restrict the HTTP calls).
 - [x] It must be possible not to send a context.
 - [x] Rebase on main.
 - [x] Check that the script cannot do any io.
 - [x] ~Introduce a `Documents.edit` action or~ require the `Documents.all` action.
 - [x] Change the `editionCode` to the clearer `function` field name in the tasks.
 - [x] Support a user provided context and maybe more (but keep function execution isolated for reproducibility).
 - [x] Support deleting documents when the `doc` is `()` (nil, null).
 - [x] Support canceling document edition.
 - [x] Multithread document edition by using rayon (and [rayon-par-bridge](https://docs.rs/rayon-par-bridge/latest/rayon_par_bridge/)).
 - [x] Limit the number of instruction by function execution.
 - [ ] ~Expose the limit of instructions in the settings.~ Not sure, in fact.
 - [x] Ignore unmodified documents in the tasks count.
 - [x] Make the `filter` field optional (not forced to be `null`).

Co-authored-by: Clément Renault <clement@meilisearch.com>
 2024-07-11 09:02:55 +00:00
Clément Renault
6e80364c50 Apply review comments 2024-07-11 11:00:27 +02:00
meili-bors[bot]
f36f34c2f7 Merge #4717 
4717: Implement intersection at end on the search pipeline r=Kerollmops a=Kerollmops

This PR is akin to #4713 and #4682 because it uses the new RoaringBitmap method to do the intersections directly on the serialized bytes for the bytes LMDB/heed returns. More work related to this issue can be done, and I listed that in #4780.

Running the following command shows where we use bitand/intersection operations and where we can potentially apply this optimization.
```sh
rg --type rust --vimgrep '\s&[=\s]' milli/src/search
```

Co-authored-by: Clément Renault <clement@meilisearch.com>
 2024-07-10 15:01:33 +00:00
Clément Renault
3bac22fd87 We do not do intersections with the universe when it is related to cache 2024-07-10 16:49:36 +02:00
Clément Renault
ce61cb7fe6 Simplify and speedup an intersection pass 2024-07-10 16:49:36 +02:00
Clément Renault
1693d1a311 Simplify the check to decide to stop a loop 2024-07-10 16:49:36 +02:00
Clément Renault
febea735ca Remove the unused universe parameter from resolve_negative_phrases 2024-07-10 16:49:36 +02:00
Clément Renault
93ba051094 Remove the invalid get_phrases_docids universe parameter 2024-07-10 16:49:35 +02:00
Clément Renault
cd7a20fa32 Make it work by avoid storing invalid stuff in the cache 2024-07-10 16:49:35 +02:00
Clément Renault
41f51adbec Do less useless intersections 2024-07-10 16:49:35 +02:00
Clément Renault
0ca1a4e805 Always do the intersections with the universe 2024-07-10 16:49:34 +02:00
Clément Renault
50a7393c55 Modify the compute_query_term_subset_docids function to accept the universe 2024-07-10 16:49:34 +02:00
Clément Renault
837274f853 Restrict even more the Rhai engine 2024-07-10 16:30:18 +02:00
Clément Renault
487997f6ad Support the new editDocumentsByFunction experimental feature 2024-07-10 16:29:18 +02:00
Clément Renault
94809090a3 Support not specifying a context 2024-07-10 16:29:18 +02:00
Clément Renault
01144b2c74 Make the edit documents by function route experimental 2024-07-10 16:29:18 +02:00
Clément Renault
e97600eead Improve the analytics for the document edition by function 2024-07-10 16:29:18 +02:00
Clément Renault
767553519d Create errors for the HTTP route issues 2024-07-10 16:29:18 +02:00
Clément Renault
aace587dd1 Create errors for the internal processing ones 2024-07-10 16:29:18 +02:00
Clément Renault
e706023969 Fix some analytics issues 2024-07-10 16:29:17 +02:00
Clément Renault
bcd0c5f5a4 Support DocumentEdition in dumps 2024-07-10 16:29:17 +02:00
Clément Renault
f35d6710f3 Update rhai to v1.19.0 2024-07-10 16:29:17 +02:00
Clément Renault
b7b8f564c3 delete-me: Simply support generating dump 2024-07-10 16:29:05 +02:00
Clément Renault
862d49e4af Editing documents requires the documents.all action (add, get, and del) 2024-07-10 16:29:05 +02:00
Clément Renault
81ec0abad1 Use the new rayon-par-bridge library 2024-07-10 16:29:04 +02:00
Clément Renault
b67d385cf0 Parallelize the edition functions 2024-07-10 16:28:54 +02:00
Clément Renault
dfecb25814 Disable the time package 2024-07-10 16:28:37 +02:00
Clément Renault
2eae2015d7 Support aborting documents edition by function 2024-07-10 16:28:15 +02:00
Clément Renault
33fa17bf12 Support deleting documents with functions 2024-07-10 16:28:15 +02:00
Clément Renault
400e6b93ce Support user-provided context for documents edition 2024-07-10 16:28:15 +02:00
Clément Renault
f32e6c32fc Rename editionCode to function 2024-07-10 16:28:15 +02:00
Clément Renault
f4add93043 Limit the number of script operations 2024-07-10 16:28:14 +02:00
Clément Renault
f07256971a Fix tests 2024-07-10 16:28:14 +02:00
Clément Renault
2fae96ac14 Show the actual number of actually edited documents 2024-07-10 16:28:14 +02:00
Clément Renault
246f0e7130 Make the filter field really optional 2024-07-10 16:28:14 +02:00
Clément Renault
45af18ae9c Check the Rhai syntax before accepting the script 2024-07-10 16:28:13 +02:00
Clément Renault
2d97164d9f It works perfectly with some Rhai 2024-07-10 16:28:13 +02:00
Clément Renault
efc156a4a4 Executing Lua works correctly 2024-07-10 16:27:36 +02:00