Commit Graph

104 Commits

Author SHA1 Message Date
0848635efc Change Actions enum to bitflags 2024-11-26 16:56:53 +02:00
9c1e54a2c8 Move crates under a sub folder to clean up the code 2024-10-21 08:18:43 +02:00
3f6301dbc9 fix the missing embedder name in the error message when trying to disable the binary quantization 2024-09-19 12:00:58 +02:00
2b6952eda1 rename the ArroyReader to an ArroyWrapper since it can read and write 2024-09-19 12:00:58 +02:00
cc45e264ca implement the binary quantization in meilisearch 2024-09-19 12:00:56 +02:00
3c5e363554 Remove default embedders 2024-09-17 16:30:43 +02:00
f18e9cb7b3 Change openai default model 2024-09-09 13:09:35 +02:00
ed19b7c3c3 Only reindex if the size increased 2024-09-03 12:07:59 +02:00
1ac008926b Add maxBytes parameter 2024-09-03 12:07:15 +02:00
21296190a3 Reindex embedders 2024-09-02 13:00:53 +02:00
5aa6cb3600 Specialize authorized error message depending on config source 2024-07-31 15:03:44 +02:00
9b7764575b openai: don't pass apiKey when it is empty 2024-07-31 15:03:44 +02:00
553440632e Introduce Setting::some_or_not_set 2024-07-25 12:01:52 +02:00
7a347966da Allow explicit dimensions for ollama 2024-07-25 12:01:51 +02:00
4654d51e05 Add custom headers for REST embedder 2024-07-25 12:01:51 +02:00
4b74803dae Change parameters in vector settings 2024-07-24 14:34:17 +02:00
d731fa661b ollama and openai use new EmbedderOptions 2024-07-24 14:34:17 +02:00
a1beddd5d9 rest embedder: use json_template 2024-07-24 14:34:17 +02:00
4109182ca4 Add json_template module 2024-07-24 14:34:12 +02:00
1a297c048e Error changes 2024-07-24 14:34:12 +02:00
303e601b87 HuggingFace: Clearer error message when a model is not supported 2024-07-23 15:13:22 +02:00
24240934f9 Improve errors when indexing documents with a user provided embedder 2024-07-16 13:39:01 +02:00
f4c94ac57f manual embedders: limit max size of errors to 250 2024-07-16 13:39:01 +02:00
4087a88dbe rest|ollama|openai: increase tries to 10 + randomize retry duration 2024-07-16 13:39:00 +02:00
5adacf2f45 OpenAI: embed only the first MAX_TOKENS tokens 2024-07-16 13:39:00 +02:00
65d0c32aa7 Allow overriding OpenAI's url 2024-07-16 13:39:00 +02:00
0a40a98bb6 Make milli use edition 2021 (#4770)
* Make milli use edition 2021

* Add lifetime annotations to milli.

* Run cargo fmt
2024-07-09 17:25:39 +02:00
ce08dc509b add more tests and improve the location of the error 2024-06-27 11:51:45 +02:00
1daaed163a Make _vectors.:embedding.regenerate mandatory + tests + error messages 2024-06-27 11:04:58 +02:00
e35ef31738 Small changes following review 2024-06-13 14:20:48 +02:00
3bc8f81abc user_provided => regenerate 2024-06-12 18:12:20 +02:00
d0b05ae691 Add EmbedderAction to settings 2024-06-12 14:50:54 +02:00
e9bf4eb100 Reformulate ParsedVectorsDiff in terms of VectorState 2024-06-12 14:11:44 +02:00
b368105272 Add EmbedderConfigs::into_inner 2024-06-12 14:11:44 +02:00
31a793d226 fix the regeneration of the embeddings in the search 2024-06-06 11:39:29 +02:00
d85ab23b82 rename all occurences of user_defined to user_provided for consistency 2024-06-06 11:39:29 +02:00
b7349910d9 implements mor review comments 2024-06-06 11:39:29 +02:00
b867829ef1 remove useless dbg 2024-06-06 11:39:29 +02:00
5d50850e12 always push the user defined vectors in arroy 2024-06-06 11:39:29 +02:00
04f6523f3c expose a new parameter to retrieve the embedders at search time 2024-06-06 11:36:11 +02:00
84e498299b Remove the vectors from the documents database 2024-06-06 11:36:11 +02:00
d35278320e Add support functions for accessing arroy writers and readers 2024-05-28 15:27:43 +02:00
3412e7fbcf "[]" is deserialized as 0 embedding rather than 1 embedding of dim 0 2024-05-22 12:25:21 +02:00
16037e2169 Don't remove embedders that are not in the config from the document DB 2024-05-22 12:24:51 +02:00
b17cb56dee Test array of vectors 2024-05-20 14:44:10 +02:00
52d9cb6e5a Refactor vector indexing
- use the parsed_vectors module
- only parse `_vectors` once per document, instead of once per embedder per document
2024-05-20 10:36:17 +02:00
98c811247e Add parsed vectors module 2024-05-20 10:25:59 +02:00
f4dd73ec8c Destructure EmbedderOptions so we don't miss some options 2024-05-02 15:39:36 +02:00
c793b6ef6d Merge #4600
4600: Fix embedders api r=ManyTheFish a=ManyTheFish

# Pull Request

## Related issue
Fixes #4594
Fixes #4595


Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-04-25 13:16:33 +00:00
d4aeff92d0 Introduce the ThreadPoolNoAbort wrapper 2024-04-24 16:40:12 +02:00