488 Commits

Author	SHA1	Message	Date
Kerollmops	15321ce924	Replace the hand-made VecDequeue by a FutureUnordered	2025-11-12 09:12:37 +01:00
Clément Renault	a097b254f8	Merge pull request #5963 from meilisearch/engprod-2128-allow-attaching-user-defined-metadata-to-tasks-and-return ... Allow to attach `customMetadata` in the document addition or update tasks	2025-11-10 15:48:46 +00:00
ManyTheFish	40e60c6f52	Fix dumpless upgrade	2025-11-10 14:03:17 +01:00
Louis Dureuil	a987d698c1	Fix error that rust-analyzer reports because it is compiling all code with the test cfg	2025-11-06 18:10:59 +01:00
Kerollmops	ce62713f02	Always create the update_files directory	2025-11-06 12:43:12 +01:00
Kerollmops	8b5d04d60f	Move the code to append a file to the tarball in a dedicated function	2025-11-06 12:07:08 +01:00
Kerollmops	1b74709b91	Extract more logic into dedicated functions	2025-11-06 12:00:25 +01:00
Kerollmops	a5c0a282c5	Better document safety problems and unwraps	2025-11-06 11:26:45 +01:00
Kerollmops	4fc048ff20	Better handle errors when trying to clone the handles	2025-11-06 11:15:34 +01:00
Kerollmops	375b5600cd	Move the tarball streaming into a dedicated function	2025-11-06 11:11:26 +01:00
Kerollmops	32b997d817	Extracts the streaming closure into an async function	2025-11-06 11:04:16 +01:00
Kerollmops	ff3090e3cc	Remove the slash-path dependency	2025-11-06 10:53:07 +01:00
Kerollmops	6c6645f945	Better seeking-to-the-start explanation	2025-11-06 10:45:47 +01:00
Clément Renault	af6473d999	Improve the comments ... Co-authored-by: Louis Dureuil <louis@meilisearch.com>	2025-11-06 10:43:14 +01:00
Kerollmops	cc4654eabd	Remove useless clippy flag	2025-11-05 16:38:05 +01:00
Kerollmops	f9d57f54df	Keep the smae naming behavior as before	2025-11-05 15:16:43 +01:00
Kerollmops	3ef1afc0f1	Introduce some backoff retries	2025-11-05 15:16:43 +01:00
Kerollmops	dbb5abebb6	Support cancelation	2025-11-05 15:16:42 +01:00
Kerollmops	700f33bd39	Clean up the code	2025-11-05 15:16:42 +01:00
Kerollmops	d01bbbccde	Support clean CLI options	2025-11-05 15:16:42 +01:00
Kerollmops	4fc506f267	Remove unused imports/code on Windows	2025-11-05 15:16:42 +01:00
Kerollmops	dc456276e5	Make clippy happy	2025-11-05 15:16:42 +01:00
Kerollmops	b2ea50cb10	Make the compression level configurable	2025-11-05 15:16:42 +01:00
Kerollmops	5074cf92ab	Disable compression entirely to avoid being CPU bound	2025-11-05 15:16:42 +01:00
Clément Renault	a92bc8d192	Improve the way we create the snapshot path	2025-11-05 15:16:42 +01:00
Clément Renault	ee538cf045	Remove useless dependencies	2025-11-05 15:16:42 +01:00
Kerollmops	2b05d63a0f	Make it finaly work but without async on the write side	2025-11-05 15:16:42 +01:00
Kerollmops	104e8918ce	Seeking the tasks/data.mdb file to the begining made the trick	2025-11-05 15:16:42 +01:00
Kerollmops	d6ec4d4f4a	Improve understanding of S3-related errors	2025-11-05 15:16:42 +01:00
Kerollmops	f0e7326b7a	Retrieve the bytesMut only when released	2025-11-05 15:16:42 +01:00
Kerollmops	c8106a0006	Fix minimum part size	2025-11-05 15:16:42 +01:00
Kerollmops	c9ab5bc0b6	Improve error messaging when missing env var	2025-11-05 15:16:42 +01:00
Clément Renault	5e0f15fd43	WIP	2025-11-05 15:16:42 +01:00
Kerollmops	4c30f090c7	WIP Do more tests	2025-11-05 15:16:42 +01:00
Clément Renault	63f247cdda	WIP sending multiparts of 250MiB	2025-11-05 15:16:42 +01:00
Clément Renault	76e4ec2168	Geenrate an async tarball	2025-11-05 15:16:42 +01:00
Kerollmops	982babdb74	WIP	2025-11-05 15:16:42 +01:00
Kerollmops	7ae2ae33d9	Make max in flights parts fro upload configurable	2025-11-05 15:16:42 +01:00
Kerollmops	cb0788ae07	Use a good mem advice for uploads	2025-11-05 15:16:42 +01:00
Kerollmops	cb3e5dc234	Move the S3 snapshots to disk into a dedicated method	2025-11-05 15:16:41 +01:00
Clément Renault	59d40a2821	Upload ten parts at a time	2025-11-05 15:16:41 +01:00
Clément Renault	98a678e73d	Use the Bytes crate to send the parts	2025-11-05 15:16:41 +01:00
Clément Renault	70292aae3c	Upload indexes under their uuids	2025-11-05 15:16:41 +01:00
Clément Renault	73521f0069	Initial working S3 uploads to RustFS	2025-11-05 15:16:41 +01:00
Louis Dureuil	4533179604	Pass tokio handle to index-scheduler	2025-11-05 15:16:41 +01:00
Louis Dureuil	083de2bfc1	Allow to attach customMetadatain the document addition or update tasks	2025-11-04 17:41:28 +01:00
Clément Renault	a63762737c	Upgrade index scheduler	2025-10-20 12:22:27 +02:00
Clément Renault	77394bd4b9	Update insta tests	2025-10-20 10:54:16 +02:00
Clément Renault	af0608ebd6	Continue to the next index if index doesn't exists	2025-10-16 16:39:51 +02:00
Clément Renault	8c7e5c094e	Improve the task batch stopped message	2025-10-16 16:39:50 +02:00