Commit Graph

10578 Commits

Author SHA1 Message Date
ff9c92c409 rename documents -> substep 2024-11-20 15:12:02 +01:00
8380ddbdcd Fix progress of into_changes 2024-11-20 15:10:09 +01:00
d4d8becfa7 Merge #5060
Some checks failed
Indexing bench (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of indexing (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for geo (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for songs (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for Wikipedia articles (push) / Run and upload benchmarks (push) Waiting to run
Publish binaries to GitHub release / Check the version validity (push) Successful in 11s
Publish binaries to GitHub release / Publish binary for ${{ matrix.os }} (meilisearch, meilisearch-macos-amd64, macos-13) (push) Waiting to run
Publish binaries to GitHub release / Publish binary for macOS silicon (meilisearch-macos-apple-silicon, aarch64-apple-darwin) (push) Waiting to run
Publish binaries to GitHub release / Publish binary for ${{ matrix.os }} (meilisearch.exe, meilisearch-windows-amd64.exe, windows-2022) (push) Failing after 21s
Publish binaries to GitHub release / Publish binary for Linux (push) Failing after 12s
Publish binaries to GitHub release / Publish binary for aarch64 (meilisearch-linux-aarch64, aarch64-unknown-linux-gnu) (push) Failing after 10s
Run the indexing fuzzer / Setup the action (push) Successful in 1h5m1s
Test suite / Tests on ubuntu-20.04 (push) Failing after 12s
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests almost all features (push) Failing after 9s
Test suite / Test disabled tokenization (push) Failing after 8s
Test suite / Run tests in debug (push) Failing after 10s
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 40s
Test suite / Run Rustfmt (push) Successful in 1m28s
Test suite / Run Clippy (push) Successful in 5m29s
5060: Batch route r=Kerollmops a=irevoire

# Pull Request

See [usage](https://www.notion.so/meilisearch/Enhance-visibility-on-batched-tasks-1194b06b651f810b8fe0fab5d72846a8).

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4977

## What does this PR do?
- For more detailed information, see the PRD.
- Added a `batchUid` to the tasks (that's the cause of all the updates of the dumps):
  - For all enqueued tasks, it's set to `None`
  - For every other tasks it must be set to something
  - ⚠️ For all the tasks imported in a dump, the `batchUid` will be set to `None` as well.
- Add two new routes:
  - `GET /batches/:uid` - to query a batch by its id
  - `GET /batches` - to retrieve a list of batches. It accepts all the same query parameters that are available on the `GET /tasks` route
- Adds new databases to query the batches directly:
  - When doing a query against the batches, the rule of thumb is that we want to return a batch iif **at least one** task in it matches the provided filter.
  - We don't need a `canceledBy` batch specific database because we can just retrieve the task and if it's a `taskCancelation` retrieve its `batchUid`
- The task cancelation has been updated and simplified a bit:
  - Instead of updating the matching tasks on disk while processing the cancelation task, we instead retrieve the task and let the `tick` function do the work afterward.
  - In the `tick` function, we now have to take care of not missing any tasks
- All the tests applied to the tasks were duplicated and updated to works with the new batches routes
- The deletion of batches doesn't contain any tests because it's already tested in the deletion of tasks (and especially highlighted in the snapshots)


Currently, one part of the PRD is not implemented: it's the progress.

Co-authored-by: Tamo <tamo@meilisearch.com>
2024-11-20 14:07:48 +00:00
867138f166 Add SP to into_changes 2024-11-20 15:07:05 +01:00
567bd4538b Fxi the into_changes stop processing 2024-11-20 14:58:25 +01:00
84600a10d1 Add MSP to document_update.into_changes() 2024-11-20 14:53:37 +01:00
35bbe1c2a2 Add failing test on settings changes 2024-11-20 14:48:12 +01:00
7d64e8dbd3 Fix Windows compilation 2024-11-20 14:40:38 +01:00
ec06879d28 apply review changes 2024-11-20 14:40:36 +01:00
83d1f858c1 Update crates/index-scheduler/src/lib.rs
Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-11-20 14:36:05 +01:00
cae8c89467 "fix" last warnings 2024-11-20 14:03:52 +01:00
a7ac590e9e implements the reverse query parameter for the batches 2024-11-20 13:29:52 +01:00
7cb8732b45 Introduce a new bincode internal error 2024-11-20 13:23:11 +01:00
8ad68dd708 stop leaking the update files of the canceled tasks 2024-11-20 13:17:54 +01:00
fe5d50969a Fix filed selector in extrators 2024-11-20 13:16:44 +01:00
56c7c5d5f0 Fix comments 2024-11-20 13:16:44 +01:00
4cdfdddd6d Fix one more 2024-11-20 13:16:43 +01:00
2afa33011a Fix tokenize_document 2024-11-20 13:16:43 +01:00
61feca1f41 More tests pass 2024-11-20 13:16:43 +01:00
f893b5153e Don't mark [""] as empty facet 2024-11-20 13:16:42 +01:00
ca779c21f9 facets: Handle boolean and skip empty strings 2024-11-20 13:16:42 +01:00
477077bdc2 Remove _vectors from fid map when there are no vectors in sight 2024-11-20 13:16:42 +01:00
b1f8aec348 Fix index_documents_check_exists_database 2024-11-20 13:16:41 +01:00
ba7f091db3 Use tokenizer on numbers and booleans 2024-11-20 13:16:41 +01:00
8049df125b Add depth to facet extraction so that null inside an array doesn't mark the entire field as null 2024-11-20 13:16:40 +01:00
50d1bd01df We no longer index geo lat and lng 2024-11-20 13:16:40 +01:00
a28d4f5d0c Fix setup_search_index_with_criteria 2024-11-20 13:16:40 +01:00
fc14f4bc66 Attempt to fix setup_search_index_with_criteria 2024-11-20 13:16:39 +01:00
5f8a82d6f5 Improve test 2024-11-20 13:16:39 +01:00
fe04e51a49 One more 2024-11-20 13:16:38 +01:00
01b27e40ad Fix a bit of the placeholder search tests 2024-11-20 13:16:38 +01:00
8076d98544 Fix stats_should_not_return_deleted_documents 2024-11-20 13:16:37 +01:00
9e951baad5 One more test passing 2024-11-20 13:16:37 +01:00
52f2fc4c46 Fail in case of user error in tests 2024-11-20 13:16:37 +01:00
3957917e0b Correctly count indexed documents 2024-11-20 13:16:36 +01:00
651c30899e Allow fetching embedders from inside tests 2024-11-20 13:16:36 +01:00
2c7a7fe4e8 Count the number of documents correctly 2024-11-20 13:16:35 +01:00
23f0c2c29b Generate internal ids only when needed 2024-11-20 13:16:35 +01:00
6641c3f59b Remove all autogenerated tests 2024-11-20 13:16:34 +01:00
07a72824b7 Subfields of _vectors are no longer part of the fid map 2024-11-20 13:16:34 +01:00
000eb55c4e fix one 2024-11-20 13:16:34 +01:00
b4bf7ce9b0 Increase the number of readers as the indexer uses readers too 2024-11-20 13:16:33 +01:00
1aef0e4037 documents! macro accepts a single object again 2024-11-20 13:16:33 +01:00
32d0e50a75 Fix all the benchmark compilation errors 2024-11-20 13:16:32 +01:00
df5884b0c1 Fix settings test 2024-11-20 13:16:32 +01:00
9e0eb5ebb0 Removed some warnings 2024-11-20 13:16:32 +01:00
3cf1352ae1 Fix the benchmark tests 2024-11-20 13:16:31 +01:00
aba8a0e9e0 Fix some tests but not all of them 2024-11-20 13:16:31 +01:00
670aff5553 Remove useless Transform methods 2024-11-20 13:16:08 +01:00
7e379b3d14 remove useless prints 2024-11-20 12:27:12 +01:00