Commit Graph

10898 Commits

Author SHA1 Message Date
f2185438ee Merge #5308
Some checks failed
Look for flaky tests / flaky (push) Failing after 13s
SDKs tests / define-docker-image (push) Failing after 7m9s
SDKs tests / .NET SDK tests (push) Has been skipped
SDKs tests / Dart SDK tests (push) Has been skipped
SDKs tests / Go SDK tests (push) Has been skipped
SDKs tests / Java SDK tests (push) Has been skipped
SDKs tests / JS SDK tests (push) Has been skipped
SDKs tests / PHP SDK tests (push) Has been skipped
SDKs tests / Python SDK tests (push) Has been skipped
SDKs tests / Ruby SDK tests (push) Has been skipped
SDKs tests / Rust SDK tests (push) Has been skipped
SDKs tests / Swift SDK tests (push) Has been skipped
SDKs tests / meilisearch-js-plugins tests (push) Has been skipped
SDKs tests / meilisearch-rails tests (push) Has been skipped
SDKs tests / meilisearch-symfony tests (push) Has been skipped
Publish binaries to GitHub release / Check the version validity (push) Successful in 12s
Publish binaries to GitHub release / Publish binary for Linux (push) Failing after 16s
Publish binaries to GitHub release / Publish binary for aarch64 (meilisearch-linux-aarch64, aarch64-unknown-linux-gnu) (push) Failing after 19s
Test suite / Tests almost all features (push) Failing after 1s
Test suite / Test with Ollama (push) Failing after 7s
Test suite / Tests on ubuntu-20.04 (push) Failing after 15s
Test suite / Test disabled tokenization (push) Failing after 9s
Test suite / Run tests in debug (push) Failing after 10s
Test suite / Run Rustfmt (push) Failing after 9s
Test suite / Run Clippy (push) Failing after 18s
Test suite / Tests on macos-13 (push) Has been cancelled
Test suite / Tests on windows-2022 (push) Has been cancelled
Publish binaries to GitHub release / Publish binary for macos-13 (push) Has been cancelled
Publish binaries to GitHub release / Publish binary for windows-2022 (push) Has been cancelled
Publish binaries to GitHub release / Publish binary for macOS silicon (meilisearch-macos-apple-silicon, aarch64-apple-darwin) (push) Has been cancelled
5308: Ollama Integration Tests r=dureuill a=Kerollmops

This PR improves test coverage of #4757 by providing a new CI to test the Ollama setup with Ollama.

## To Do
- [x] Clean up the commits
- [x] Feature gate the Ollama tests and run them only in the CI

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
2025-02-06 17:21:51 +00:00
ca1ad51564 Put the Ollama tests under a feature 2025-02-06 17:27:47 +01:00
a1d1e7c82a Setup dedicated CI to run the Ollama tests 2025-02-06 17:12:17 +01:00
56438bdea4 Introduce an Ollama integration test 2025-02-06 17:12:17 +01:00
e93a5719ef Merge #5293
Some checks failed
Test suite / Tests on ubuntu-20.04 (push) Failing after 1s
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Run tests in debug (push) Failing after 14s
Test suite / Tests on windows-2022 (push) Failing after 24s
Test suite / Run Clippy (push) Failing after 31s
Test suite / Run Rustfmt (push) Successful in 1m45s
Run the indexing fuzzer / Setup the action (push) Successful in 1h5m3s
Indexing bench (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of indexing (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for geo (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for songs (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for Wikipedia articles (push) / Run and upload benchmarks (push) Has been cancelled
Test suite / Tests on macos-13 (push) Has been cancelled
5293: Support merging update and replacement operations r=irevoire a=Kerollmops

This PR fixes #5286 by modifying the auto-batcher and how we merge documents when preparing them for the new indexer.

## To do
- [x] Make sure we can auto-batch different operation types.
- [x] Make sure the indexer correctly understands and mixes the different kinds.
- [x] Create a test to see if it mixes the documents correctly.
- [x] Modify the auto-batcher tests for the new behavior.


Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
2025-02-03 11:28:41 +00:00
d34f0b606c Update crates/milli/src/update/new/document_change.rs 2025-02-03 12:08:52 +01:00
acc400face Support merging update and replacement operations 2025-02-03 11:47:17 +01:00
aa2327591e Add more mixing updates and replacements tests 2025-02-03 10:34:07 +01:00
a6f9e0ddf0 Fix auto batching related tests 2025-02-03 10:34:07 +01:00
60470bb647 Fix the tests to use the new replace/update documents 2025-02-03 10:34:07 +01:00
294e1ba16d Fix functions calls to use the new mixed system 2025-02-03 10:34:06 +01:00
8e6893ddbe Make sure we correctly mix different document operations 2025-02-03 10:34:06 +01:00
d018346f18 Make the auto-batcher batche replacement with updates 2025-02-03 10:34:05 +01:00
de98656ed1 Merge #5210
Some checks failed
Test suite / Tests almost all features (push) Has been skipped
Run the indexing fuzzer / Setup the action (push) Failing after 6s
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 14s
Test suite / Run tests in debug (push) Failing after 2s
Test suite / Run Rustfmt (push) Failing after 8s
Test suite / Tests on windows-2022 (push) Failing after 20s
Test suite / Run Clippy (push) Successful in 5m48s
Indexing bench (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of indexing (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for geo (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for songs (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for Wikipedia articles (push) / Run and upload benchmarks (push) Has been cancelled
Test suite / Tests on macos-13 (push) Has been cancelled
5210: Improve test performance of get_index.rs  r=irevoire a=DerTimonius

# Pull Request

## Related issue
related to #4840

## What does this PR do?
This PR aims to improve the performance of the tests in `get_index.rs`.

There is a small issue though: 
the `list_multiple_indexes` test works great when ran alone, but when running with other tests it fails with a `corrupted task queue` error. I guess this has something to do with using a shared server, but I was not really able to pinpoint the issue.

Also, the `no_index_return_empty_list` does not work a shared server (as there now will always be at least one index on the server) and I was not really sure if rebuilding the whole suite for `get_and_paginate_indexes` should be viable? While waiting for feedback on the issue mentioned above, I'll try to change the `get_and_paginate_indexes` test so that it can use the shared server

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Timon Jurschitsch <timon.jurschitsch@gmail.com>
Co-authored-by: Timon Jurschitsch <103483059+DerTimonius@users.noreply.github.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
2025-01-27 10:04:08 +00:00
da7469be38 removed unrelated files 2025-01-27 10:35:34 +01:00
df9d10ac44 Merge #5284
5284: Fix [5281] Removed CouldNotUpgrade from error file  r=irevoire a=manojks1999

# Pull Request

## Related issue
Fixes #5281

## What does this PR do?
- ...

## PR checklist
Please check if your PR fulfills the following requirements:
- [ * ] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [ * ] Have you read the contributing guidelines?
- [ * ] Have you made sure that the title is accurate and descriptive of the changes?


Co-authored-by: manojks1999 <9743manoj@gmail.com>
v1.13.0-rc.0
2025-01-27 09:26:39 +00:00
528d9d6d8b Removed CouldNotUpgrade from error file 2025-01-26 21:04:57 +05:30
4fb5c39b92 resolve merge conflicts 2025-01-24 14:35:54 +01:00
022205af90 Merge #5279
Some checks failed
Test suite / Tests on ubuntu-20.04 (push) Failing after 0s
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Run tests in debug (push) Failing after 2s
Test suite / Tests on windows-2022 (push) Failing after 22s
Test suite / Run Rustfmt (push) Successful in 2m14s
Test suite / Run Clippy (push) Successful in 5m21s
Run the indexing fuzzer / Setup the action (push) Successful in 1h4m54s
Test suite / Tests on macos-13 (push) Has been cancelled
Indexing bench (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of indexing (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for geo (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for songs (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for Wikipedia articles (push) / Run and upload benchmarks (push) Has been cancelled
5279: Bring back changes from v1.12.7 into main r=dureuill a=Kerollmops

This PR brings back v1.12.7 into main.

Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
2025-01-24 11:48:46 +00:00
50280bf02b Support offline upgrade up to v1.12.7 2025-01-24 12:25:33 +01:00
9b579069df Comment the max grant of the bbqueue
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-24 12:18:32 +01:00
f5a4a1c8b2 Give more RAM to bbqueue.
- bbqueue buffers used to have (5% * 2%) / num_threads
- they now have 5% / num_threads
2025-01-24 12:18:32 +01:00
5ab4cdb1f3 Reduce the maximum grant possible we can store in the BBQueue 2025-01-24 12:18:32 +01:00
1f54f07f72 Merge #5264
Some checks failed
Indexing bench (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of indexing (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for geo (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for songs (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for Wikipedia articles (push) / Run and upload benchmarks (push) Waiting to run
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 12s
Test suite / Run tests in debug (push) Failing after 2s
Test suite / Tests on windows-2022 (push) Failing after 21s
Test suite / Run Rustfmt (push) Failing after 8s
Test suite / Run Clippy (push) Successful in 6m30s
Run the indexing fuzzer / Setup the action (push) Successful in 1h5m21s
Test suite / Tests on macos-13 (push) Has been cancelled
5264: Dumpless upgrade r=dureuill a=irevoire

# Pull Request
Usage: https://meilisearch.notion.site/Dumpless-upgrade-fff4b06b651f81f1acafe24d4687b3f7?pvs=74

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5162

## What does this PR do?
- Implement the dumpless upgrade with multiple hooks:
  - In meilisearch directly before the task queue has been opened
  - In the index-scheduler while processing the task
  - In milli while upgrading the indexes
- There is no hook at search/query time to handle the old version of a database. That's left to the next person upgrading a database
- A new special type of task (`upgradeDatabase`) that can be retried has been introduced
- A new experimental cli flag has been introduced
- The version has been upgraded to the v1.13.0 in this PR otherwise it was a lot of useless work to test the dumpless upgrade
- Multiple tests have been introduced

## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Update the issue template we use for features, mentioning what we should do in case of a database upgrade
- [ ] The experimental feature discussion should be opened and updated in the PR
- [ ] Update the PRD
    - [ ] Add the new error codes
    - [ ] Add the task details
    - [ ] Add the telemetry

## Notes

The new tests introduced are not _that_ slow
![image](https://github.com/user-attachments/assets/c5884540-482f-41eb-97ef-fc995c62d666)



Co-authored-by: Tamo <tamo@meilisearch.com>
2025-01-23 16:22:37 +00:00
73d8a4eace Remove db.snapshot 2025-01-23 17:21:42 +01:00
c1e5897076 Do not assume v1.12 when there is no index-scheduler version 2025-01-23 17:16:53 +01:00
718a98fbbf remove : char from filenames 2025-01-23 17:08:35 +01:00
86bf231d29 Change to meilitool after rebase 2025-01-23 16:59:32 +01:00
182c3f4b80 Write assumed version to the index-scheduler version db when it is missing 2025-01-23 16:51:25 +01:00
c1eba66443 introduce a corruption in the v1.12 data.ms field distribution 2025-01-23 16:51:24 +01:00
7197ced673 fix the bad index version on opening 2025-01-23 16:51:24 +01:00
4f21ee6c66 update the data.ms snapshot 2025-01-23 16:51:24 +01:00
787472453d write the version of the index while upgrading it 2025-01-23 16:51:24 +01:00
8f65f35de9 rewrite part of the index-scheduler upgrade test 2025-01-23 16:51:23 +01:00
c27c923439 introduce a trait to upgrade the indexes 2025-01-23 16:51:23 +01:00
fd5649091d add the upgradeTo field in the details 2025-01-23 16:51:23 +01:00
9a57736773 fix the early exit when rewriting a batch 2025-01-23 16:51:23 +01:00
7740997ea8 reintroduce the unrecoverable error and use it where its supposed to be used 2025-01-23 16:51:22 +01:00
7eb23f73ba add the version to the index-scheduler snapshots + fix a bug when opening an index scheduler for the first time 2025-01-23 16:51:22 +01:00
b9e9fc376a add the version in the index-scheduler 2025-01-23 16:51:22 +01:00
27bf2f1298 remove the empty progress made for the upgrade database 2025-01-23 16:51:22 +01:00
d4d82fbd0c commit the index wtxn before the index-scheduler wtxn 2025-01-23 16:51:21 +01:00
eda09a54da improve the index-scheduler tests 2025-01-23 16:51:21 +01:00
b132d70413 fix the details in all cases 2025-01-23 16:51:21 +01:00
e41ebd3047 expose the number of database in the index-scheduler and rewrite the lib.rs to use the value provided in the options instead of a magic number 2025-01-23 16:51:21 +01:00
705d31e8bd apply all the comments changes 2025-01-23 16:51:21 +01:00
7d95950ce6 fix warning 2025-01-23 16:51:21 +01:00
c6b4c21c23 update the snapshots after the rebase 2025-01-23 16:51:20 +01:00
bf96fdb858 update the cli url 2025-01-23 16:51:20 +01:00
41eeffd88d fmt 2025-01-23 16:51:20 +01:00