71d53f413f
increase the margin allowed to delete task
2024-12-03 11:07:03 +01:00
054622bd16
Merge #5094
...
5094: Implement a bbqueue channel between the extractors and the writer r=dureuill a=Kerollmops
This PR switches from a bounded crossbeam channel only with allocated entries for the communication between the extractors and the writer to a [BBQueue](https://github.com/jamesmunns/bbqueue )-based system with a Single Producer Single Consumer kind of Circular/Ring Buffers channel.
- [x] Implement the BBQueue channel system...
- [x] with a crossbeam channel to wake up the receiver.
- [x] Manage the BBQueue allocated memory dynamically.
- [x] Support content that doesn't fit in the bbqueues.
Co-authored-by: Clément Renault <clement@meilisearch.com >
2024-12-03 08:00:55 +00:00
beeb31ce41
Update crates/index-scheduler/src/lib.rs
2024-12-02 15:32:16 +01:00
a439fa3e1a
While spamming the batches route we could see a processing batch becoming missing and then finished, this commit ensures the batches goes from processing to finished directly
2024-12-02 12:02:16 +01:00
e1e76f39d0
Clean up dependencies
2024-11-27 14:30:34 +01:00
36962b943b
First batch of PR comment
2024-11-21 16:38:11 +01:00
1f9692cd04
Increase map size for tests
2024-11-20 17:52:21 +01:00
6e6acfcf1b
Merge branch 'main' into indexer-edition-2024
2024-11-20 16:59:58 +01:00
83d1f858c1
Update crates/index-scheduler/src/lib.rs
...
Co-authored-by: Clément Renault <clement@meilisearch.com >
2024-11-20 14:36:05 +01:00
a7ac590e9e
implements the reverse query parameter for the batches
2024-11-20 13:29:52 +01:00
8ad68dd708
stop leaking the update files of the canceled tasks
2024-11-20 13:17:54 +01:00
7e379b3d14
remove useless prints
2024-11-20 12:27:12 +01:00
e145d71a62
implements the two last TODOs
2024-11-20 10:51:06 +01:00
b906e3ed70
improve the way we access the mutex
2024-11-20 10:51:06 +01:00
4abcd9c04e
add some stats on the batches
2024-11-20 10:51:06 +01:00
1fcb9526f5
fix the task cancelation
2024-11-20 10:51:06 +01:00
15eefa4fcc
fixes a lot of small issue, the test about the cancellation is still failing
2024-11-20 10:51:05 +01:00
ad9763ffcd
copy multiple task query tests to batches. Currently, they fails
2024-11-20 10:49:25 +01:00
d489f5635f
add the mapping between the task and batches
2024-11-20 10:49:23 +01:00
a1251c3c83
Implements the get all batches route with filters working
2024-11-20 10:42:55 +01:00
6062914654
add the batch_id to the tasks
2024-11-20 10:42:54 +01:00
bfefaf71c2
Progress displayed in logs
2024-11-19 09:32:52 +01:00
5f93651cef
fixes
2024-11-18 16:23:11 +01:00
e9d17136b2
Add deadline of 3 seconds to embedding requests made in the context of hybrid search
2024-11-18 12:15:11 +01:00
c202f3dbe2
fix tests and revert change in behavior when primary_key_from_op != primary_key_from_db && index.is_empty()
2024-11-18 10:59:05 +01:00
677d7293f5
Fix a lot of primary key related tests
2024-11-18 10:59:05 +01:00
1fcd5f091e
Remove progress from task
2024-11-12 12:23:13 +01:00
2eb1801e85
reverse the order of the task queue
2024-11-07 19:19:44 +01:00
ee03743355
Merge branch 'indexer-edition-2024' into indexer-edition-2024-doc-chunks
2024-11-06 15:50:53 +01:00
cf6ad1ae5e
Merge branch 'main' into tmp-release-v1.11.0
2024-11-04 16:14:44 +01:00
9c1e54a2c8
Move crates under a sub folder to clean up the code
2024-10-21 08:18:43 +02:00
