|
|
2d8f9a9849
|
Add support for back/slashes
|
2019-11-11 21:23:08 +01:00 |
|
|
|
de2b8672d4
|
Make the tokenizer understand strange whitespaces/quotes
|
2019-11-04 16:41:58 +01:00 |
|
|
|
9dce41ed6b
|
Cargo clippy pass
|
2019-10-18 13:30:06 +02:00 |
|
|
|
ca26a0f2e4
|
Cargo fmt pass
|
2019-10-18 13:30:06 +02:00 |
|
|
|
cdd69290c3
|
test: Make the tests work with new separator limits
|
2019-09-24 20:49:42 +02:00 |
|
|
|
175b3dcb75
|
fix: Do not consider underscores and middle dash hard
|
2019-09-24 20:14:20 +02:00 |
|
|
|
6b9426a051
|
feat: Support underscore as a split character
|
2019-09-24 13:56:32 +02:00 |
|
|
|
abf7191eec
|
feat: Make the Tokenizer able to support tokenizing sequences
|
2019-04-29 14:32:00 +02:00 |
|
|
|
a745819ddf
|
feat: Simplify the Tokenizer to use the LinearStrGroupBy type
|
2019-04-29 14:31:37 +02:00 |
|
|
|
19e67dcf0b
|
feat: Move query splitting into the tokenizer workspace
|
2019-04-29 14:31:37 +02:00 |
|
|
|
1897da5348
|
feat: Move tokenizer things into the meilidb-tokenizer workspace
|
2019-04-29 14:31:37 +02:00 |
|
