Kerollmops
2b0e17ede0
Make sure arroy is using the rayon thread-pool
2025-02-06 15:28:10 +01:00
meili-bors[bot]
796acd1aee
Merge #5288
...
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 13s
Test suite / Run tests in debug (push) Failing after 13s
Test suite / Run Clippy (push) Failing after 19s
Test suite / Tests on windows-2022 (push) Failing after 48s
Test suite / Run Rustfmt (push) Successful in 1m28s
Test suite / Tests on macos-13 (push) Has been cancelled
5288: Improve AI logging r=dureuill a=Kerollmops
This PR fixes #5285 and brings the changes from #5233 to simplify debugging indexation and search performance issues related to AI. The following texts can be found in the logs to debug and understand performance issues:
- `embed_one: search` represents the time we spent waiting for the embedding generation, i.e., OpenAI, local HuggingFace, Ollama.
- `filtered_universe: search::universe` the time spent filtering the documents.
- ~`next_bucket: search::vector_sort` is the time spent finding the nearest neighbors (ANNs) in the vector store (arroy), locally~ was being triggered too many times.
- `indexing::vectors` is the time arroy spends indexing the new vectors for a batch.
- `documents::extract vectors` and `documents::merge vectors` to see the time spent generating and writing the embeddings.
Co-authored-by: Kerollmops <clement@meilisearch.com >
2025-02-04 10:20:45 +00:00
Tamo
d34f0b606c
Update crates/milli/src/update/new/document_change.rs
2025-02-03 12:08:52 +01:00
Kerollmops
acc400face
Support merging update and replacement operations
2025-02-03 11:47:17 +01:00
Kerollmops
aa2327591e
Add more mixing updates and replacements tests
2025-02-03 10:34:07 +01:00
Kerollmops
60470bb647
Fix the tests to use the new replace/update documents
2025-02-03 10:34:07 +01:00
Kerollmops
8e6893ddbe
Make sure we correctly mix different document operations
2025-02-03 10:34:06 +01:00
Kerollmops
424c5bde40
Move the embedding computation and extraction log to debug
2025-01-29 16:40:36 +01:00
Kerollmops
cb1b7513af
Log the memory metrics only once
2025-01-29 15:21:52 +01:00
Clément Renault
a9d0f4a002
Improve english comments
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com >
2025-01-29 15:16:40 +01:00
Kerollmops
db032079d8
Show indexation allocated memory
2025-01-29 14:21:02 +01:00
Clément Renault
a00796c46a
Improve the naming in the log message
2025-01-29 14:21:02 +01:00
Kerollmops
6112bd8caa
Display the channel congestion
2025-01-29 14:21:02 +01:00
Kerollmops
cec88cfc29
Measure the bbqueue congestion
2025-01-29 14:21:02 +01:00
Kerollmops
4a5923a55e
log the time arroy took to insert embeddings
2025-01-27 14:22:17 +01:00
Clément Renault
9b579069df
Comment the max grant of the bbqueue
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com >
2025-01-24 12:18:32 +01:00
Louis Dureuil
f5a4a1c8b2
Give more RAM to bbqueue.
...
- bbqueue buffers used to have (5% * 2%) / num_threads
- they now have 5% / num_threads
2025-01-24 12:18:32 +01:00
Kerollmops
5ab4cdb1f3
Reduce the maximum grant possible we can store in the BBQueue
2025-01-24 12:18:32 +01:00
Tamo
787472453d
write the version of the index while upgrading it
2025-01-23 16:51:24 +01:00
Tamo
c27c923439
introduce a trait to upgrade the indexes
2025-01-23 16:51:23 +01:00
Tamo
41eeffd88d
fmt
2025-01-23 16:51:20 +01:00
Tamo
20ac59c946
fix the field distribution when upgrading from the v1_12
2025-01-23 16:51:19 +01:00
Tamo
cfc1e193b6
update the test with the stats
2025-01-23 16:51:19 +01:00
Tamo
0cc25c7e4c
add a large test importing a data.ms from the v1.12.0
2025-01-23 16:51:18 +01:00
Tamo
3ef7a478cd
move the version check to the task queue
2025-01-23 16:48:32 +01:00
Tamo
d3654906bf
Add the new tasks with most of the job done
2025-01-23 16:48:32 +01:00
Louis Dureuil
d6063079af
Unify facet strings by their normalized value
2025-01-22 15:50:42 +01:00
Louis Dureuil
a6470a0c37
Improve error log
2025-01-22 15:50:41 +01:00
Louis Dureuil
8a54f14b8e
Demote panic to error log
2025-01-22 15:49:24 +01:00
Kerollmops
63c8cbae5b
Improve the panic message when deleting an unknown entry
2025-01-14 10:31:44 +01:00
Louis Dureuil
72ded27e98
Update after review
2025-01-14 10:24:50 +01:00
Louis Dureuil
4070895a21
Add support to upgrade to v1.12.3 in meilitool
2025-01-14 10:24:27 +01:00
Louis Dureuil
a21711f473
Fix test
2025-01-14 10:23:59 +01:00
Louis Dureuil
f0ec8cbffe
Add currently failing test
2025-01-14 10:23:15 +01:00
meili-bors[bot]
247eaed872
Merge #5221
...
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 1m4s
Test suite / Tests on ubuntu-20.04 (push) Failing after 27s
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Run tests in debug (push) Failing after 20s
Test suite / Run Clippy (push) Successful in 8m45s
Test suite / Run Rustfmt (push) Successful in 2m31s
5221: Merge bitmaps by using `Extend::extend` r=Kerollmops a=Kerollmops
This PR tries to speed up the merging of bitmaps by using [the new `Extend::extend` implementation](https://github.com/RoaringBitmap/roaring-rs/pull/306 ).
Co-authored-by: Clément Renault <clement@meilisearch.com >
2025-01-13 13:43:28 +00:00
meili-bors[bot]
cc4aca78c4
Merge #5220
...
5220: Merge back changes of v1.12.2 in main r=dureuill a=dureuill
Co-authored-by: curquiza <curquiza@users.noreply.github.com >
Co-authored-by: Tamo <tamo@meilisearch.com >
Co-authored-by: dureuill <dureuill@users.noreply.github.com >
Co-authored-by: Louis Dureuil <louis@meilisearch.com >
2025-01-13 10:54:36 +00:00
Clément Renault
00a03742ff
Prefer using extend when merging bitmaps than unions (less allocations)
2025-01-09 10:42:38 +01:00
Louis Dureuil
d11e359244
When spilling on the next fid, no longer ignore children
2025-01-09 10:36:38 +01:00
Louis Dureuil
09d45439c7
Check valid_facet_value as part of a filter of the iterator
2025-01-09 10:36:38 +01:00
Louis Dureuil
5d92da0c73
No longer ignore the first child without parent
2025-01-09 10:36:38 +01:00
Louis Dureuil
677bb39e73
Modernize valid_lmdb_key
2025-01-09 10:36:38 +01:00
Louis Dureuil
85ea77de0b
Switch to an iterative algorithm for find_changed_parents
2025-01-09 10:36:38 +01:00
Louis Dureuil
03317be0bd
Update after review
2025-01-09 10:36:38 +01:00
Louis Dureuil
4aa7c8f7b1
Remove unused FacetFieldIdOperation
2025-01-09 10:36:37 +01:00
Louis Dureuil
ce57a342a3
center groups
2025-01-09 10:36:37 +01:00
Louis Dureuil
1cc6cd78e0
Fix uselessly deep stack trace
2025-01-09 10:36:37 +01:00
Louis Dureuil
c204afdc79
Update snapshot
2025-01-09 10:36:37 +01:00
Louis Dureuil
c14967eeac
Use new incremental facet indexing and enable sanity checks in debug
2025-01-09 10:36:35 +01:00
Louis Dureuil
f38db86120
Add new incremental facet indexing
2025-01-09 10:24:36 +01:00
Louis Dureuil
50b155fa2d
add valid_facet_value utility function
2025-01-09 10:24:36 +01:00
