473 Commits

Author	SHA1	Message	Date
Kerollmops	cbc0878158	Introduce some backoff retries	2025-11-04 17:46:01 +01:00
Kerollmops	8967e40083	Support cancelation	2025-11-04 16:30:02 +01:00
Kerollmops	f6a0db9cb8	Clean up the code	2025-11-04 14:28:03 +01:00
Kerollmops	b64b083003	Support clean CLI options	2025-11-04 14:28:03 +01:00
Kerollmops	eab2b806e9	Remove unused imports/code on Windows	2025-11-04 09:07:59 +01:00
Kerollmops	f7d9425c59	Remove unused imports/code on Windows	2025-11-03 18:52:58 +01:00
Kerollmops	f7f68f81a4	Make it run on Windows as we use a standard io::pipe	2025-11-03 18:19:54 +01:00
Kerollmops	6e0d0d306b	Make clippy happy	2025-11-03 17:45:29 +01:00
Kerollmops	34dc67aea8	Make the compression level configurable	2025-10-24 11:47:47 +02:00
Kerollmops	fbe56822f0	Disable compression entirely to avoid being CPU bound	2025-10-23 13:32:32 +02:00
Clément Renault	5e1af30b42	Improve the way we create the snapshot path	2025-10-20 16:48:14 +02:00
Clément Renault	d298b21a95	Remove useless dependencies	2025-10-20 16:48:14 +02:00
Kerollmops	a4ad87febf	Make it finaly work but without async on the write side	2025-10-20 16:48:14 +02:00
Kerollmops	2caa2be441	Seeking the tasks/data.mdb file to the begining made the trick	2025-10-20 16:48:14 +02:00
Kerollmops	a829ded023	Improve understanding of S3-related errors	2025-10-20 16:48:14 +02:00
Kerollmops	98be43b66b	Retrieve the bytesMut only when released	2025-10-20 16:48:14 +02:00
Kerollmops	3ec4750426	Fix minimum part size	2025-10-20 16:48:14 +02:00
Kerollmops	d717ec3486	Improve error messaging when missing env var	2025-10-20 16:48:14 +02:00
Clément Renault	0c06bdefac	WIP	2025-10-20 16:48:13 +02:00
Kerollmops	62a8133bcd	WIP Do more tests	2025-10-20 16:48:13 +02:00
Clément Renault	c1f7542dfa	WIP sending multiparts of 250MiB	2025-10-20 16:48:13 +02:00
Clément Renault	59e2394e69	Geenrate an async tarball	2025-10-20 16:48:13 +02:00
Kerollmops	430bc91c4c	WIP	2025-10-20 16:48:13 +02:00
Kerollmops	1172dce093	Make max in flights parts fro upload configurable	2025-10-20 16:48:13 +02:00
Kerollmops	21140a33e6	Use a good mem advice for uploads	2025-10-20 16:48:13 +02:00
Kerollmops	9ed7a81495	Move the S3 snapshots to disk into a dedicated method	2025-10-20 16:48:13 +02:00
Clément Renault	c8c1d95efd	Upload ten parts at a time	2025-10-20 16:48:13 +02:00
Clément Renault	10d1e26478	Use the Bytes crate to send the parts	2025-10-20 16:48:13 +02:00
Clément Renault	cd781d267b	Upload indexes under their uuids	2025-10-20 16:48:13 +02:00
Clément Renault	b16af4a763	Initial working S3 uploads to RustFS	2025-10-20 16:48:13 +02:00
Louis Dureuil	1514d13ab3	Pass tokio handle to index-scheduler	2025-10-20 16:48:13 +02:00
Clément Renault	a63762737c	Upgrade index scheduler	2025-10-20 12:22:27 +02:00
Clément Renault	77394bd4b9	Update insta tests	2025-10-20 10:54:16 +02:00
Clément Renault	af0608ebd6	Continue to the next index if index doesn't exists	2025-10-16 16:39:51 +02:00
Clément Renault	8c7e5c094e	Improve the task batch stopped message	2025-10-16 16:39:50 +02:00
Clément Renault	c064737137	Remove duplicated logic in auto batching of tasks	2025-10-16 16:33:20 +02:00
Clément Renault	1d188a7ad3	Make the compaction tasks a priority over the export ones	2025-10-16 13:01:23 +02:00
Kerollmops	ddc76ad0dc	Delete the leftover compaction files from canceled operations	2025-10-15 16:49:25 +02:00
Kerollmops	ffacf1c002	Introduce the new IndexMapper index path method	2025-10-15 16:49:25 +02:00
Kerollmops	5a49b93b77	Use constant tempfile name to reuse tempfile	2025-10-15 16:49:25 +02:00
Kerollmops	1d701c6980	Fix upgrade tests	2025-10-13 10:40:15 +02:00
Kerollmops	0203adb9cb	Add a no-op when upgrading the index scheduler	2025-10-13 10:28:31 +02:00
Louis Dureuil	14de657d36	Use the "currently_processing_index" to avoid potentially blocking the search during compaction	2025-10-08 15:45:38 +02:00
Kerollmops	9a36c090bf	Do not return the EnvClosingEvent	2025-10-08 15:38:45 +02:00
Kerollmops	3aca010b42	Recompute the stats	2025-10-08 15:33:12 +02:00
Clément Renault	f358538f4f	Improve the pre-compaction size information	2025-10-08 15:33:12 +02:00
Clément Renault	9068857ba1	Make the tests pass	2025-10-08 15:33:12 +02:00
Clément Renault	d241157084	Make Clippy happy	2025-10-08 15:33:12 +02:00
Clément Renault	69f73b1d74	Introduce a function to effectively close an index	2025-10-08 15:33:12 +02:00
Clément Renault	202794f620	Expose the env closing event so we can wait for the index to close	2025-10-08 15:33:12 +02:00