eada6de261
Divide threshold by ten
2024-03-04 18:02:54 +01:00
9d1f489a37
Fix facet incremental indexing
2024-02-21 18:42:16 +01:00
48026aa75c
fix PR comments
2024-02-13 15:19:01 +01:00
55de96f74e
Update milli/src/update/facet/mod.rs
...
Co-authored-by: Clément Renault <clement@meilisearch.com >
2024-02-13 14:22:10 +01:00
39c83cb3d9
fix clippy
2024-02-12 09:12:54 +01:00
be1b054b05
Compute chunk size based on the input data size ant the number of indexing threads
2024-02-08 17:28:37 +01:00
e773dfa9ba
get rids of log in milli and add logs for the bucket sort
2024-02-08 15:04:05 +01:00
d32eb11329
Move to the v0.20.0-alpha.9 of heed
2023-11-27 11:52:22 +01:00
0dbf1a16ff
Make clippy happy
2023-11-23 14:11:38 +01:00
462b4c0080
Fix the tests
2023-11-23 12:07:35 +01:00
0d4482625a
Make the changes to use heed v0.20-alpha.6
2023-11-23 11:43:58 +01:00
1bccf2079e
Correctly mark non-tests as non-tests
2023-11-06 11:03:56 +01:00
b1d1355b69
remove tests on soft-deleted
2023-10-31 16:36:27 +01:00
03ddb4f310
use deladd in facet update tests
2023-10-31 16:36:27 +01:00
290e773d23
remove more warnings and fix some tests
2023-10-30 11:41:22 +01:00
113527f466
Remove soft-deleted related methods from Index
2023-10-30 11:41:22 +01:00
2263dff02b
Stop using removed delete pipelines almost everywhere
2023-10-30 11:41:22 +01:00
d651b3ef01
Remove delete documents files
2023-10-30 11:41:20 +01:00
073f89db79
Fix facet tests
2023-10-30 11:40:20 +01:00
59f88c14b3
Simplify facet update after removing Index::faceted_documents_ids
2023-10-30 11:39:29 +01:00
14832cb324
Remove Index::faceted_documents_ids
2023-10-30 11:37:32 +01:00
f67ff3a738
Facets Bulk update
2023-10-30 11:36:40 +01:00
0c47defeee
Work on fid docid facet values rewrite
2023-10-30 11:29:06 +01:00
d772073dfa
use a bufreader everytime there is a grenad<file>
2023-10-10 15:00:30 +02:00
35758db9ec
Truncate the the normalized long facets used in search for facet value
2023-08-08 16:38:30 +02:00
df528b41d8
Normalize for the search the facets values
2023-07-20 17:57:07 +02:00
e9a3029c30
Use the right field id to write the string facet values FST
2023-06-28 15:01:51 +02:00
f36de2115f
Make clippy happy
2023-06-28 15:01:50 +02:00
15a4c05379
Store the facet string values in multiple FSTs
2023-06-28 14:58:41 +02:00
c4a40e7110
Use the writemap flag to reduce the memory usage
2023-05-15 10:15:33 +02:00
1b78231e18
Make clippy happy
2023-01-17 18:25:54 +01:00
ad9937c755
Fix tests after adding DeletionStrategy
2022-12-19 10:07:17 +01:00
f2cf981641
Add more tests and allow disabling of soft-deletion outside of tests
...
Also allow disabling soft-deletion in the IndexDocumentsConfig
2022-12-05 10:51:01 +01:00
f198b20c42
Add facet deletion tests that use both the incremental and bulk methods
...
+ update deletion snapshots to the new database format
2022-10-26 13:47:46 +02:00
2295e0e3ce
Use real delete function in facet indexing fuzz tests
...
By deleting multiple docids at once instead of one-by-one
2022-10-26 13:47:46 +02:00
acc8caebe6
Add link to GitHub PR to document of update/facet module
2022-10-26 13:47:46 +02:00
a034a1e628
Move StrRefCodec and ByteSliceRefCodec to their own files
2022-10-26 13:47:46 +02:00
1165ba2171
Make facet deletion incremental
2022-10-26 13:47:04 +02:00
3baa34d842
Fix compiler errors/warnings
2022-10-26 13:47:04 +02:00
86d9f50b9c
Fix bugs in incremental facet indexing with variable parameters
...
e.g. add one facet value incrementally with a group_size = X and then
add another one with group_size = Y
It is not actually possible to do so with the public API of milli,
but I wanted to make sure the algorithm worked well in those cases
anyway.
The bugs were found by fuzzing the code with fuzzcheck, which I've added
to milli as a conditional dev-dependency. But it can be removed later.
2022-10-26 13:47:04 +02:00
985a94adfc
cargo fmt
2022-10-26 13:47:04 +02:00
27454e9828
Document and refine facet indexing algorithms
2022-10-26 13:47:04 +02:00
bee3c23b45
Add comparison benchmark between bulk and incremental facet indexing
2022-10-26 13:47:04 +02:00
b2f01ad204
Refactor facet database tests
2022-10-26 13:47:04 +02:00
9026867d17
Give same interface to bulk and incremental facet indexing types
...
+ cargo fmt, oops, sorry for the bad history :(
2022-10-26 13:47:04 +02:00
330c9eb1b2
Rename facet codecs and refine FacetsUpdate API
2022-10-26 13:47:04 +02:00
485a72306d
Refactor facet-related codecs
2022-10-26 13:47:04 +02:00
9b55e582cd
Add FacetsUpdate type that wraps incremental and bulk indexing methods
2022-10-26 13:47:04 +02:00
e8a156d682
Reorganise facets database indexing code
2022-10-26 13:46:46 +02:00
