snowykami/meilisearch
1
0
Fork 0
mirror of https://github.com/meilisearch/meilisearch.git synced 2025-10-30 23:46:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
10,950 Commits 257 Branches 544 Tags
2b0e17ede0a113f5694d3407431eef394ca60663
Commit Graph

10950 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Louis Dureuil
0e68718027 Add detailed spans 2024-07-31 13:05:47 +02:00
Louis Dureuil
7c3fc8c655 Split settings and document facet string extractions 2024-07-31 10:57:46 +02:00
Louis Dureuil
8acd3f50bb skip normalization when the locales and values are the same 2024-07-31 09:53:00 +02:00
meili-bors[bot]
25791e3f46 Merge #4836 
4836: Attach declared localized-attributes subroutes r=dureuill a=dureuill

RC.0 unexpectedly doesn't contain the `GET /indexes/{indexUid}/localized-attributes` and `PUT /indexes/{indexUid}/localized-attributes` subroute.

This PR makes them available.

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
v1.10.0-rc.1 		2024-07-30 19:01:54 +00:00
meili-bors[bot]
866922ecc3 Merge #4808 
4808: Make the tests run faster r=irevoire a=irevoire

## Index-Scheduler

### Only check the consistency of the index-scheduler on snapshots when running in release mode

This saves 12s on the tests, and since the tests run in release mode in the CI, we don't lose any information.
From 28s to 16s

### We were snapshotting the index for no reason in `advance_till`, I removed this call

This saved an additional 8s on the tests, going from 16s to 8s.

----

After these two optimizations, the test suite as a whole executes 14% quicker

## Meilisearch integration tests

While profiling this test suite, nothing stands out. The only noticeable thing is that we're losing most of our time creating and dropping threads.
I made the theory that by sharing a single common instance between all integrations tests I would gain some time again.

In 355a7acd1c I saved another 15s by only testing this theory on the module that tests the error messages. 
But we can do it on many more tests. **We must take care of not making any test flaky, though**.

## Use two indexing threads

By moving from one to two indexing threads, we gain an additional 30% in performance.

# Conclusion

## Before

The execution of the test suite was taking around:
- 4m40s on my computer
- 15 minutes on the debug CI with cache
- 29 minutes on the Windows CI with cache

## After

The execution of the test suite is taking around:
- 2m20 on my computer
- 8 minutes on the debug CI with cache
- 29 minutes on the Windows CI with cache

## This means the test suite should now run ~50% faster on your computer; the CI may report errors twice faster, but we'll still wait for ~the same amount of time to merge a PR


Co-authored-by: Tamo <tamo@meilisearch.com>
 2024-07-30 15:11:30 +00:00
Tamo
f05ea04879 In prometheus metrics return the route pattern instead of the real route when returning the HTTP requests total 2024-07-30 16:24:49 +02:00
Tamo
b1b3a1a98b add a get, set and put test for the localized attributes setting 2024-07-30 15:51:02 +02:00
meili-bors[bot]
143d6cde10 Merge #4835 
4835: Log error from main using tracing r=irevoire a=dureuill

Engine follow-up to https://github.com/meilisearch/meilisearch-support/issues/252#issuecomment-2251288276 (private link)

> `@meilisearch/engine-team` we need to open a PR to tracing::error! when an error occurs in the Meilisearch main. It would be nice to have it included in the second RC

<img width="1349" alt="Error logged when launching Meilisearch to import dump on path where the dump doesn't exist" src="https://github.com/user-attachments/assets/e5d2ae6e-f810-4029-9787-3b6ea9d47cfd">

---

<img width="1349" alt="Error logges when launching Meilisearch with a db path that is not writeable" src="https://github.com/user-attachments/assets/f672d78d-04b0-4d02-9402-259eaa6e2b62">



Co-authored-by: Louis Dureuil <louis@meilisearch.com>
 2024-07-30 13:43:50 +00:00
Tamo
c457069367 ensure a test is 100% not flaky 2024-07-30 15:41:51 +02:00
Tamo
bb1283222e make clippy happy 2024-07-30 15:10:56 +02:00
Tamo
7a5a38f870 fix a sync issue on empty indexes 2024-07-30 15:09:12 +02:00
Tamo
ded3cd0dd6 an additionnal 30% of perf for the tests 2024-07-30 15:03:54 +02:00
Tamo
68f885f1c4 fix two snapshots 2024-07-30 14:45:59 +02:00
Tamo
9372c34dab prepare the tests to share indexes with api key 2024-07-30 14:34:11 +02:00
Tamo
6666c57880 reduce the number of thread spawned by milli 2024-07-30 14:34:10 +02:00
Tamo
b53a019b07 fix the initialization problem over the shared indexes with documents 2024-07-30 14:24:57 +02:00
Tamo
d262b1df32 craft an API over the Shared Server and Shared index to avoid hard to debug mistakes 2024-07-30 14:24:57 +02:00
Tamo
ed795bc837 fmt 2024-07-30 14:24:57 +02:00
Tamo
993264227d reuse an index with already indexed documents instead of reindexing from scratch 2024-07-30 14:24:57 +02:00
Tamo
953d3a44bd make the new_shared function synchronous and stop indexing documents when it's not required 2024-07-30 14:24:57 +02:00
Tamo
e5345fb0eb shave off 15s by providing a shared instance to the integration tests 2024-07-30 14:24:55 +02:00
Tamo
2d9a055fb9 stops snapshotting in advance_till when we don't need to 2024-07-30 13:57:12 +02:00
Tamo
110dc01f40 only check the consistency of the index-scheduler on snapshots when running in release mode 2024-07-30 13:57:12 +02:00
Louis Dureuil
9719dec443 Attach declared attributes-localized subroutes 2024-07-29 16:19:35 +02:00
Louis Dureuil
fa77a949aa Log error from main using tracing 2024-07-29 14:58:39 +02:00
meili-bors[bot]
abe128476f Merge #4830 
4830: Use the dtolnay's Rust Toolchain r=dureuill a=Kerollmops

Fixes the CI by using another rust-toolchain GitHub repo.

Note: the [helix-editor/rust-toolchain repository](https://github.com/helix-editor/rust-toolchain) has been deleted so we moved to the [dtolnay/rust-toolchain](https://github.com/dtolnay/rust-toolchain) one. However, the dtolnay's one doesn't support `rust-toolchain.toml` and the version is directly in the rust-toolchain@version. We keep the `rust-toolchain.toml` for local builds only.

Co-authored-by: Clément Renault <clement@meilisearch.com>
v1.10.0-rc.0 		2024-07-29 08:33:59 +00:00
Clément Renault
a663e408ad Move to the right rust toolchain version 2024-07-29 10:06:34 +02:00
Clément Renault
986991277f Use the dtolnay rust toolchain 2024-07-29 10:00:40 +02:00
meili-bors[bot]
c2c1ba39ee Merge #4826 
4826: Update Charabia v0.9.0 r=dureuill a=ManyTheFish

# Pull Request

## Related Changelog
https://github.com/meilisearch/charabia/releases/tag/v0.9.0

## Notable Change for Meilisearch
Adds all math symbols from https://www.compart.com/en/unicode/category/Sm to the default separator list.



Co-authored-by: ManyTheFish <many@meilisearch.com>
 2024-07-25 14:08:38 +00:00
ManyTheFish
35567b2137 Update Charabia v0.9.0 2024-07-25 16:02:14 +02:00
meili-bors[bot]
00c97c7152 Merge #4818 
4818: Custom headers and QoL improvements r=ManyTheFish a=dureuill

# Pull Request

## Related issue
Fixes #4734 
Depends on #4815 

## What does this PR do?
- Adds custom headers for rest embedders ([public usage](https://meilisearch.notion.site/v1-10-AI-search-changes-737c9d7d010d4dd685582bf5dab579e2#41354652885242c899def07e36a66d49))
- Quality of life: allow specifying `dimensions` for `ollama` embedders ([public usage](https://meilisearch.notion.site/v1-10-AI-search-changes-737c9d7d010d4dd685582bf5dab579e2#37218531431343dab3d2d3a9a1937e9d)). As for `rest` embedders, specifying `dimensions` disables the "test" embedding when the embedder is spawned.
- Improve error message again when indexing documents that don't have a vector for a user-provided vector
  1. Remove the contents of the document
  2. Display the docid of the first document that triggered the error
  3. Indicate how many documents in that chunk suffered from the same issue for that embedder


Co-authored-by: Louis Dureuil <louis@meilisearch.com>
 2024-07-25 13:33:11 +00:00
Louis Dureuil
d4ea7cc2a9 fix clippy 👉👈 2024-07-25 12:10:32 +02:00
Louis Dureuil
8532fe8afc Fix tests 2024-07-25 12:10:32 +02:00
Louis Dureuil
2413592bbf Display docid when there are documents without manual embeddings for a manual embedder 2024-07-25 12:10:32 +02:00
Louis Dureuil
553440632e Introduce Setting::some_or_not_set 2024-07-25 12:01:52 +02:00
Louis Dureuil
7a347966da Allow explicit dimensions for ollama 2024-07-25 12:01:51 +02:00
Louis Dureuil
6c598fa06d test custom headers 2024-07-25 12:01:51 +02:00
Louis Dureuil
8338df0dbe Fix tests 2024-07-25 12:01:51 +02:00
Louis Dureuil
4654d51e05 Add custom headers for REST embedder 2024-07-25 12:01:51 +02:00
Louis Dureuil
22ef2d877f Ensure test server has a single indexing thread 2024-07-25 12:01:51 +02:00
meili-bors[bot]
76bc2c18e8 Merge #4819 
4819: Language settings r=dureuill a=ManyTheFish

# Pull Request

## Related issue
Fixes #4749 

## What does this PR do?
- [Implement localized search](c0c6955c0d)
- [Implement localized attributes settings](bde827b055)

## Related PRD

- [PRD](https://www.notion.so/meilisearch/Define-language-settings-to-impact-relevancy-bee62e18b7584c4f87d18a7654855329)
- [Public usage](https://www.notion.so/meilisearch/v1-10-Language-settings-usage-26c5d98b553349d9abacbe7aff698e4e)


Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
 2024-07-25 09:00:33 +00:00
Louis Dureuil
59115fd058 Fix tests 2024-07-25 10:52:57 +02:00
ManyTheFish
a918561ac1 Fix PR comments 2024-07-25 10:52:56 +02:00
ManyTheFish
70d71581ee fix clippy 2024-07-25 10:52:56 +02:00
ManyTheFish
4fbe048cbf Update Cargo.lock 2024-07-25 10:52:56 +02:00
ManyTheFish
e06fbcc607 Update snapshots 2024-07-25 10:52:56 +02:00
ManyTheFish
04fa44e7eb Implement localized attributes settings 2024-07-25 10:51:27 +02:00
ManyTheFish
90c0a6db7d Implement localized search 2024-07-25 10:51:27 +02:00
ManyTheFish
d82f8fd904 Add tests 2024-07-25 10:51:27 +02:00
ManyTheFish
cc02920f2b Update charabia 2024-07-25 10:51:27 +02:00