cd74ed300c
Use Hannoy instead of arroy
2025-08-14 11:53:34 +02:00
b086c51a23
new settings indexer
2025-07-02 00:05:13 +02:00
f8232976ed
Implement in new document indexer
2025-07-02 00:05:12 +02:00
e414284335
Clippy too many arguments
2025-06-30 14:25:28 +02:00
6db5939f84
Re-integrate embedder stats
2025-06-30 09:52:06 +02:00
d35b2d8d33
minor fixes
2025-06-30 09:52:06 +02:00
900be0ccad
Extract or regenerate vectors related to settings changes
2025-06-26 18:14:48 +02:00
e60b855a54
Delete embedders from arroy
2025-06-26 18:14:48 +02:00
e704f4d1ec
Reimplement reindexing shell
2025-06-26 18:14:48 +02:00
29f6eeff8f
Remove lots of Arcs
2025-06-26 12:15:08 +02:00
d7721fe607
Format
2025-06-24 12:20:22 +02:00
4925b30196
Move embedder stats out of progress
2025-06-23 15:24:14 +02:00
8b23eddc10
Dumpless upgrade
2025-04-30 18:03:50 +02:00
db7ce03763
Improve the performances of computing the size of the documents database
2025-03-26 17:40:12 +01:00
009c36a4d0
Add support for the progress API of arroy
2025-03-13 19:00:43 +01:00
5ef7767429
Let arroy uses all the memory available instead of 50% of the 70%
2025-03-13 15:06:03 +01:00
ef9d9f8481
set the memory in arroy
2025-03-13 11:29:00 +01:00
9f3663e768
Implement Incremental document database stats computing
2025-02-26 17:01:35 +01:00
05cc8c650c
Expose the write channel congestion in the batches
2025-02-19 15:47:54 +01:00
0c3e7fe963
Merge #5316
...
Test suite / Tests on ubuntu-20.04 (push) Failing after 2s
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Run tests in debug (push) Failing after 16s
Test suite / Run Clippy (push) Failing after 12s
Test suite / Run Rustfmt (push) Failing after 32s
Test suite / Tests on macos-13 (push) Has been cancelled
Test suite / Tests on windows-2022 (push) Has been cancelled
5316: Fix the dumpless upgrade corruption r=dureuill a=irevoire
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5280
## What does this PR do?
- Add a test that ensure we write the version in the index-scheduler even if we have a bug while writing the VERSION file
- Do what was described in the issue
Co-authored-by: Tamo <tamo@meilisearch.com >
2025-02-10 09:53:57 +00:00
45f843ccb9
fmt
2025-02-10 10:46:42 +01:00
2b0e17ede0
Make sure arroy is using the rayon thread-pool
2025-02-06 15:28:10 +01:00
cb1b7513af
Log the memory metrics only once
2025-01-29 15:21:52 +01:00
db032079d8
Show indexation allocated memory
2025-01-29 14:21:02 +01:00
f5a4a1c8b2
Give more RAM to bbqueue.
...
- bbqueue buffers used to have (5% * 2%) / num_threads
- they now have 5% / num_threads
2025-01-24 12:18:32 +01:00
4275833bab
Rename compute.rs to post_process.rs
2025-01-07 15:31:20 +01:00
de7f8c4406
refactor indexer mod
2025-01-07 15:29:02 +01:00
fa885e75b4
rename the send_progress in progress
2024-12-11 18:13:12 +01:00
867e6a8f1d
rename the send_progress field to progress since it s not sending anything
2024-12-11 16:25:01 +01:00
df9b68f8ed
inital implementation of the progress
2024-12-11 16:25:01 +01:00
a751972c57
Prefer using a stable than a random hash builder
2024-12-10 14:25:53 +01:00
89637bcaaf
Use bumparaw-collections in Meilisearch/milli
2024-12-10 11:52:20 +01:00
f5dd8dfc3e
Rollback max memory usage changes
2024-12-09 10:26:30 +01:00
3a11e39c01
Force max_memory to a min of 100MiB
2024-12-04 17:53:30 +01:00
8ecb726683
Fix the minimun BBQueue channel threshold
2024-12-03 15:49:11 +01:00
d5c07ef7b3
Manage key length conversion error correctly
2024-12-02 11:03:00 +01:00
30eb0e5b5b
Rename recv and read methods to recv_action and recv_frame
2024-12-02 10:08:01 +01:00
14ee7aa84c
Make sure the BBQueue is at least 50 MiB
2024-11-28 18:02:48 +01:00
8a35cd1743
Adjust the BBQueue buffers to use 2% instead of 10%
2024-11-28 16:00:15 +01:00
3c7ac093d3
Take the BBQueue capacity into account in the max memory
2024-11-28 15:43:14 +01:00
b57dd5c58e
Remove the Vector variant and use the Vectors
2024-11-28 15:20:43 +01:00
096a28656e
Fix a bug around deleting all the vectors of a doc
2024-11-28 15:15:06 +01:00
cc4bd54669
Correctly construct the Embeddings struct
2024-11-28 13:53:25 +01:00
58eab9a018
Send large payload through crossbeam
2024-11-28 12:01:06 +01:00
e83534a430
Fix the indexer::index to correctly use the rayon::ThreadPool
2024-11-27 16:27:43 +01:00
98d4a2909e
Fix the way we spawn the rayon threadpool
2024-11-27 16:05:44 +01:00
cc63802115
Modify and return the IndexEmbeddings to write them later
2024-11-27 14:58:03 +01:00
acec45ad7c
Send a WakeUp when writing data in the BBQueue buffers
2024-11-27 14:33:23 +01:00
70802eb7c7
Fix most issues with the lifetimes
2024-11-27 14:32:42 +01:00
6ac5b3b136
Finish most of the channels types
2024-11-27 14:32:26 +01:00
