snowykami/meilisearch
1
0
Fork 0
mirror of https://github.com/meilisearch/meilisearch.git synced 2025-10-31 16:06:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,577 Commits 257 Branches 544 Tags
952fabf8a0452dc2be01540a6b8286cc048e1334
Commit Graph

278 Commits

Author SHA1 Message Date
Tamo
1f54dfa883 update the macro to look more like an enum 2024-12-11 16:26:09 +01:00
Tamo
786b0fabea implement the progress for almost all the tasks 2024-12-11 16:26:08 +01:00
Tamo
26733c705d add progress for the task deletion and task cancelation 2024-12-11 16:25:02 +01:00
Tamo
ab75f53efd update all snapshots 2024-12-11 16:25:02 +01:00
Tamo
df9b68f8ed inital implementation of the progress 2024-12-11 16:25:01 +01:00
Kerollmops
bb00e70087 Reintroduce the document addition logs 2024-12-11 10:39:04 +01:00
Kerollmops
6b269795d2 Update bumparaw-collections to 0.1.2 2024-12-10 14:25:13 +01:00
Kerollmops
89637bcaaf Use bumparaw-collections in Meilisearch/milli 2024-12-10 11:52:20 +01:00
Louis Dureuil
866ac91be3 Fix error messages 2024-12-10 11:06:58 +01:00
airycanon
b75f1f4c17 fix tests 
# Conflicts:
#	crates/index-scheduler/src/batch.rs
#	crates/index-scheduler/src/snapshots/lib.rs/fail_in_process_batch_for_document_deletion/after_removing_the_documents.snap
#	crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fifth_task_succeeds.snap
#	crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fourth_task_fails.snap
#	crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/second_task_fails.snap
#	crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/third_task_fails.snap
#	crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key_batch_wrong_key/second_and_third_tasks_fails.snap
#	crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/all_other_tasks_succeeds.snap
#	crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/second_task_fails.snap
#	crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/third_task_succeeds.snap

# Conflicts:
#	crates/index-scheduler/src/batch.rs
#	crates/meilisearch/src/search/mod.rs
#	crates/meilisearch/tests/vector/mod.rs

# Conflicts:
#	crates/index-scheduler/src/batch.rs
 2024-12-06 02:03:02 +08:00
airycanon
95ed079761 attach index name in errors 
# Conflicts:
#	crates/index-scheduler/src/batch.rs

# Conflicts:
#	crates/index-scheduler/src/batch.rs
#	crates/meilisearch/src/search/mod.rs
 2024-12-06 01:12:13 +08:00
meili-bors[bot]
26be5e0733 Merge #5123 
5123: Fix batch details r=dureuill a=irevoire

# Pull Request

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5079
Fixes https://github.com/meilisearch/meilisearch/issues/5112

## What does this PR do?
- Make the processing tasks actually processing in the stats of the batch instead of enqueued
- Stop counting one extra task for all non-prioritized batches in the stats
- Add a test

Co-authored-by: Tamo <tamo@meilisearch.com>
 2024-12-05 15:21:55 +00:00
Tamo
95975944d7 fix the dumps missing the empty swap index tasks 2024-12-05 14:23:38 +01:00
Tamo
7a2af06b1e update the impacted snapshots 2024-12-04 15:52:24 +01:00
Tamo
cb0c3a5aad stop adding one enqueued tasks to all unprioritized batches 2024-12-04 15:48:28 +01:00
Tamo
cbcf6c9ba3 make the processing tasks as processing in a batch 2024-12-04 14:48:48 +01:00
Tamo
bf742d81cf add a test 2024-12-04 14:47:02 +01:00
Tamo
71d53f413f increase the margin allowed to delete task 2024-12-03 11:07:03 +01:00
meili-bors[bot]
054622bd16 Merge #5094 
5094: Implement a bbqueue channel between the extractors and the writer r=dureuill a=Kerollmops

This PR switches from a bounded crossbeam channel only with allocated entries for the communication between the extractors and the writer to a [BBQueue](https://github.com/jamesmunns/bbqueue)-based system with a Single Producer Single Consumer kind of Circular/Ring Buffers channel.

 - [x] Implement the BBQueue channel system...
 - [x] with a crossbeam channel to wake up the receiver.
 - [x] Manage the BBQueue allocated memory dynamically.
 - [x] Support content that doesn't fit in the bbqueues.

Co-authored-by: Clément Renault <clement@meilisearch.com>
 2024-12-03 08:00:55 +00:00
meili-bors[bot]
2e879c1df8 Merge #5109
Some checks failed
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 11s
Test suite / Run tests in debug (push) Failing after 11s
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 24s
Test suite / Run Rustfmt (push) Successful in 1m22s
Test suite / Run Clippy (push) Successful in 6m29s
5109: Fix autobatch r=dureuill a=dureuill

Fixes most SDK tests and flaky failures

Changes:

- Make sure that the settings are not autobatched with document operations, as the new indexer no longer supports this operating mode

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
 2024-12-02 16:30:51 +00:00
Tamo
beeb31ce41 Update crates/index-scheduler/src/lib.rs 2024-12-02 15:32:16 +01:00
Louis Dureuil
057143214d Fix warnings 2024-12-02 14:42:31 +01:00
Louis Dureuil
6a1d26a60c Update autobatching tests 2024-12-02 14:15:15 +01:00
Louis Dureuil
d78f4666a0 Fix autobatching of documents and settings 2024-12-02 12:25:01 +01:00
Tamo
a439fa3e1a While spamming the batches route we could see a processing batch becoming missing and then finished, this commit ensures the batches goes from processing to finished directly 2024-12-02 12:02:16 +01:00
Clément Renault
98d4a2909e Fix the way we spawn the rayon threadpool 2024-11-27 16:05:44 +01:00
Clément Renault
e1e76f39d0 Clean up dependencies 2024-11-27 14:30:34 +01:00
meili-bors[bot]
d0b2c0a523 Merge #5091
Some checks failed
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 11s
Test suite / Run tests in debug (push) Failing after 10s
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 39s
Test suite / Run Rustfmt (push) Successful in 1m38s
Test suite / Run Clippy (push) Successful in 23m11s
5091: Settings opt out r=Kerollmops a=ManyTheFish

# Pull Request

Related PRD: https://www.notion.so/meilisearch/API-usage-Settings-to-opt-out-indexing-features-fff4b06b651f8108ade3f858aeb16b14?pvs=4

## Related issue
Fixes #4979 

- [x] Add setting opt-out
- [x] Add analytics
- [x] Add tests


Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
 2024-11-26 15:50:28 +00:00
Louis Dureuil
fa15be5bc4 Add span around commit 2024-11-26 09:45:48 +01:00
ManyTheFish
d66dc363ed Test and implement settings opt-out 2024-11-25 18:23:22 +01:00
meili-bors[bot]
19e6f675b3 Merge #4900 
4900: Indexer edition 2024 r=Kerollmops a=dureuill

This PR is implementing the indexer edition 2024, largely inspired by [the ideas from this blog post](https://blog.kerollmops.com/meilisearch-is-too-slow).

Fixes https://github.com/meilisearch/meilisearch/issues/4985

## Features
- Stream-first approach to reading documents.
- Minimum disk write operations.
- RAM usage-first approach to avoid modifying common bitmaps on disk but in memory.
- Reduced LMDB fragmentation by writing entries only once...
- ...computing the final version of the entries in parallel...
- ...and storing them in write-optimized data structures before sending them to the BTree (LMDB).
- Indexing in multiple transactions to improve large dataset support (dumps).


Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
 2024-11-21 16:19:10 +00:00
Louis Dureuil
221e547e86 Slight changes 2024-11-21 16:47:44 +01:00
ManyTheFish
36962b943b First batch of PR comment 2024-11-21 16:38:11 +01:00
Many the fish
ff38f29981 Update crates/index-scheduler/src/batch.rs 
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
 2024-11-21 14:18:39 +01:00
Louis Dureuil
1f9692cd04 Increase map size for tests 2024-11-20 17:52:21 +01:00
Tamo
1e694ae432 improve the count of the number of tasks in a batch 2024-11-20 17:48:26 +01:00
Tamo
71807cac6d makes clippy happy 2024-11-20 17:40:58 +01:00
Tamo
21a2264782 improve the details and stats of the current batch processing 2024-11-20 17:25:55 +01:00
Louis Dureuil
6e6acfcf1b Merge branch 'main' into indexer-edition-2024 2024-11-20 16:59:58 +01:00
Louis Dureuil
ff9c92c409 rename documents -> substep 2024-11-20 15:12:02 +01:00
Clément Renault
8380ddbdcd Fix progress of into_changes 2024-11-20 15:10:09 +01:00
Louis Dureuil
84600a10d1 Add MSP to document_update.into_changes() 2024-11-20 14:53:37 +01:00
Tamo
ec06879d28 apply review changes 2024-11-20 14:40:36 +01:00
Tamo
83d1f858c1 Update crates/index-scheduler/src/lib.rs 
Co-authored-by: Clément Renault <clement@meilisearch.com>
 2024-11-20 14:36:05 +01:00
Tamo
a7ac590e9e implements the reverse query parameter for the batches 2024-11-20 13:29:52 +01:00
Tamo
8ad68dd708 stop leaking the update files of the canceled tasks 2024-11-20 13:17:54 +01:00
Tamo
7e379b3d14 remove useless prints 2024-11-20 12:27:12 +01:00
Tamo
bdb51a85fe now that the task cancelation shares their started at with all the tasks of their batch we don't need the trick of retrieving the previous batch anymore 2024-11-20 10:51:07 +01:00
Tamo
e145d71a62 implements the two last TODOs 2024-11-20 10:51:06 +01:00
Tamo
d9a4e69990 push a missing snapshot 2024-11-20 10:51:06 +01:00