snowykami/meilisearch
1
0
Fork 0
mirror of https://github.com/meilisearch/meilisearch.git synced 2025-07-26 00:01:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
9,692 Commits 240 Branches 478 Tags
29b44e55410453bc4d903b52a1245dcafccbb8f4
Commit Graph

9692 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
meili-bors[bot]
29b44e5541 Merge #4626 
4626: Edit Documents with Rhai r=ManyTheFish a=Kerollmops

This PR introduces a first version of [the _Update Documents with Function_ (internal)](https://www.notion.so/meilisearch/Update-Documents-by-Function-45f87b13e61c4435b73943768a490808). It uses [the Rhai programming language](https://rhai.rs/) to let users express the modifications they want apply.

You can read more about the way to use this functions on [the Usage PRD Page](https://meilisearch.notion.site/Edit-Documents-with-Rhai-0cff8fea7655436592e7c8a6de932062?pvs=25). The [prototype is available](https://github.com/meilisearch/meilisearch/actions/runs/9038384483) through Docker by using the following command:

```
docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-edit-documents-with-rhai-3
```

## TODO
 - [x] Support the `DocumentEdition` task in dumps.
 - [x] Remove the unwraps and panics.
 - [x] Improve error codes for the `function` parameter.
 - [x] [Update Rhai to v1.19.0](https://github.com/rhaiscript/rhai/releases/tag/v1.19.0) 🚀
 - [x] Make it an experimental feature (only restrict the HTTP calls).
 - [x] It must be possible not to send a context.
 - [x] Rebase on main.
 - [x] Check that the script cannot do any io.
 - [x] ~Introduce a `Documents.edit` action or~ require the `Documents.all` action.
 - [x] Change the `editionCode` to the clearer `function` field name in the tasks.
 - [x] Support a user provided context and maybe more (but keep function execution isolated for reproducibility).
 - [x] Support deleting documents when the `doc` is `()` (nil, null).
 - [x] Support canceling document edition.
 - [x] Multithread document edition by using rayon (and [rayon-par-bridge](https://docs.rs/rayon-par-bridge/latest/rayon_par_bridge/)).
 - [x] Limit the number of instruction by function execution.
 - [ ] ~Expose the limit of instructions in the settings.~ Not sure, in fact.
 - [x] Ignore unmodified documents in the tasks count.
 - [x] Make the `filter` field optional (not forced to be `null`).

Co-authored-by: Clément Renault <clement@meilisearch.com>
 2024-07-11 09:02:55 +00:00
Clément Renault
6e80364c50 Apply review comments 2024-07-11 11:00:27 +02:00
meili-bors[bot]
f36f34c2f7 Merge #4717 
4717: Implement intersection at end on the search pipeline r=Kerollmops a=Kerollmops

This PR is akin to #4713 and #4682 because it uses the new RoaringBitmap method to do the intersections directly on the serialized bytes for the bytes LMDB/heed returns. More work related to this issue can be done, and I listed that in #4780.

Running the following command shows where we use bitand/intersection operations and where we can potentially apply this optimization.
```sh
rg --type rust --vimgrep '\s&[=\s]' milli/src/search
```

Co-authored-by: Clément Renault <clement@meilisearch.com>
 2024-07-10 15:01:33 +00:00
Clément Renault
3bac22fd87 We do not do intersections with the universe when it is related to cache 2024-07-10 16:49:36 +02:00
Clément Renault
ce61cb7fe6 Simplify and speedup an intersection pass 2024-07-10 16:49:36 +02:00
Clément Renault
1693d1a311 Simplify the check to decide to stop a loop 2024-07-10 16:49:36 +02:00
Clément Renault
febea735ca Remove the unused universe parameter from resolve_negative_phrases 2024-07-10 16:49:36 +02:00
Clément Renault
93ba051094 Remove the invalid get_phrases_docids universe parameter 2024-07-10 16:49:35 +02:00
Clément Renault
cd7a20fa32 Make it work by avoid storing invalid stuff in the cache 2024-07-10 16:49:35 +02:00
Clément Renault
41f51adbec Do less useless intersections 2024-07-10 16:49:35 +02:00
Clément Renault
0ca1a4e805 Always do the intersections with the universe 2024-07-10 16:49:34 +02:00
Clément Renault
50a7393c55 Modify the compute_query_term_subset_docids function to accept the universe 2024-07-10 16:49:34 +02:00
Clément Renault
837274f853 Restrict even more the Rhai engine 2024-07-10 16:30:18 +02:00
Clément Renault
487997f6ad Support the new editDocumentsByFunction experimental feature 2024-07-10 16:29:18 +02:00
Clément Renault
94809090a3 Support not specifying a context 2024-07-10 16:29:18 +02:00
Clément Renault
01144b2c74 Make the edit documents by function route experimental 2024-07-10 16:29:18 +02:00
Clément Renault
e97600eead Improve the analytics for the document edition by function 2024-07-10 16:29:18 +02:00
Clément Renault
767553519d Create errors for the HTTP route issues 2024-07-10 16:29:18 +02:00
Clément Renault
aace587dd1 Create errors for the internal processing ones 2024-07-10 16:29:18 +02:00
Clément Renault
e706023969 Fix some analytics issues 2024-07-10 16:29:17 +02:00
Clément Renault
bcd0c5f5a4 Support DocumentEdition in dumps 2024-07-10 16:29:17 +02:00
Clément Renault
f35d6710f3 Update rhai to v1.19.0 2024-07-10 16:29:17 +02:00
Clément Renault
b7b8f564c3 delete-me: Simply support generating dump 2024-07-10 16:29:05 +02:00
Clément Renault
862d49e4af Editing documents requires the documents.all action (add, get, and del) 2024-07-10 16:29:05 +02:00
Clément Renault
81ec0abad1 Use the new rayon-par-bridge library 2024-07-10 16:29:04 +02:00
Clément Renault
b67d385cf0 Parallelize the edition functions 2024-07-10 16:28:54 +02:00
Clément Renault
dfecb25814 Disable the time package 2024-07-10 16:28:37 +02:00
Clément Renault
2eae2015d7 Support aborting documents edition by function 2024-07-10 16:28:15 +02:00
Clément Renault
33fa17bf12 Support deleting documents with functions 2024-07-10 16:28:15 +02:00
Clément Renault
400e6b93ce Support user-provided context for documents edition 2024-07-10 16:28:15 +02:00
Clément Renault
f32e6c32fc Rename editionCode to function 2024-07-10 16:28:15 +02:00
Clément Renault
f4add93043 Limit the number of script operations 2024-07-10 16:28:14 +02:00
Clément Renault
f07256971a Fix tests 2024-07-10 16:28:14 +02:00
Clément Renault
2fae96ac14 Show the actual number of actually edited documents 2024-07-10 16:28:14 +02:00
Clément Renault
246f0e7130 Make the filter field really optional 2024-07-10 16:28:14 +02:00
Clément Renault
45af18ae9c Check the Rhai syntax before accepting the script 2024-07-10 16:28:13 +02:00
Clément Renault
2d97164d9f It works perfectly with some Rhai 2024-07-10 16:28:13 +02:00
Clément Renault
efc156a4a4 Executing Lua works correctly 2024-07-10 16:27:36 +02:00
Clément Renault
ba85959642 Support filtering the documents to edit with lua 2024-07-10 16:23:21 +02:00
Clément Renault
1702b5cf44 Prepare for processing documents edition 2024-07-10 16:23:21 +02:00
meili-bors[bot]
2099b4f0dd Merge #4786 
4786: Update dependencies r=Kerollmops a=irevoire

# Pull Request

## Related issue
Fixes #4753

## What does this PR do?
- Update all dependencies except rustls
- [x] Release charabia
- [x] Update charabia
- [x] Double check that the docker build works after updating charabia



Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
 2024-07-10 13:23:54 +00:00
Tamo
0d5bc4578e Update CONTRIBUTING.md 
Co-authored-by: Clément Renault <clement@meilisearch.com>
 2024-07-10 15:21:43 +02:00
Tamo
8f60ad0a23 apply review comments 2024-07-10 14:38:19 +02:00
Tamo
9570139eeb update contributing.md with the new lindera update 2024-07-10 14:28:43 +02:00
Clément Renault
9d6885793e Upgrade dependencies 2024-07-10 13:46:24 +02:00
Clément Renault
98cd6a865c Update dependencies after removing useless ones 2024-07-10 13:37:24 +02:00
Clément Renault
5f4530ce57 Remove more unused dependencies 2024-07-10 13:36:34 +02:00
Tamo
0ecaf861fa fix ci 2024-07-10 10:06:59 +02:00
Tamo
4d5005b01a make clippy happy 2024-07-10 10:06:59 +02:00
Tamo
952e742321 update charabia 2024-07-09 23:41:29 +02:00