29 Commits

Author	SHA1	Message	Date
Louis Dureuil	1a3f4e719d	Vector document trait	2024-10-21 10:38:21 +02:00
Louis Dureuil	68a2502388	Introduce indexer level bumpalo	2024-10-10 22:23:05 +02:00
Clément Renault	83c09d0db0	Remove the now, useless AppendOnlyVec library	2024-10-07 16:38:45 +02:00
Clément Renault	0409a26cd8	Replace the concurrent vec by a linked list	2024-10-03 15:15:29 +02:00
Clément Renault	b7a5ba100e	Move the ParallelIteratorExt into the parallel_iterator_ext module	2024-10-01 11:11:52 +02:00
Clément Renault	0a8cb471df	Introduce the AppendOnlyVec struct for the parallel computing	2024-10-01 11:11:25 +02:00
ManyTheFish	bb7a503e5d	Compute prefix databases ... We are now computing the prefix FST and a prefix delta in the Merger thread, after all the databases are written, the main thread will recompute the prefix databases based on the prefix delta without needing any grenad temporary file anymore	2024-10-01 09:57:06 +02:00
Clément Renault	86d5e6d9ff	Use the new Lru	2024-09-25 14:54:56 +02:00
Clément Renault	759b9b1546	Introduce a new custom Lru	2024-09-25 14:49:12 +02:00
ManyTheFish	3f7a500f3b	Build prefix fst	2024-09-25 14:36:06 +02:00
Clément Renault	e0c7067355	Expose an IndexedParallelIterator to the index function	2024-09-24 17:24:59 +02:00
Clément Renault	8287c2644f	Support CSV again	2024-09-10 21:10:28 +01:00
ManyTheFish	c50d3edc4a	Integrate first searchable exctrator	2024-09-03 11:02:39 +02:00
Clément Renault	bcb1aa3d22	Find a temporary solution to par into iter on an HashMap ... Spoiler: Do not use an HashMap but drain it into a Vec	2024-09-02 19:39:48 +02:00
Clément Renault	72e7b7846e	Renaming the indexers	2024-09-02 14:42:27 +02:00
Clément Renault	e639ec79d1	Move the indexers into their own modules	2024-09-02 10:42:19 +02:00
Clément Renault	bb885a5810	Fix the merge for roaring bitmap	2024-09-01 23:20:19 +02:00
Clément Renault	b625d31c7d	Introduce the PartialDumpIndexer indexer that generates document ids in parallel	2024-08-30 15:07:21 +02:00
Clément Renault	271ce91b3b	Add the rayon Threadpool to the index function parameter	2024-08-30 14:34:24 +02:00
Clément Renault	54f2eb4507	Remove duplication of grenad merger	2024-08-30 14:34:05 +02:00
Clément Renault	794ebcd582	Replace grenad with the new grenad various-improvement branch	2024-08-30 11:53:59 +02:00
Clément Renault	b7c77c7a39	Use the latest version of the obkv crate	2024-08-30 11:53:59 +02:00
Clément Renault	0c57cf7565	Replace obkv with the temporary new version of it	2024-08-30 11:53:58 +02:00
Clément Renault	27df9e6c73	Introduce the indexer::index function that runs the indexation	2024-08-30 11:53:58 +02:00
Clément Renault	45c060831e	Introduce typed channels and the merger loop	2024-08-30 11:53:58 +02:00
Clément Renault	874c1ac538	First channels types	2024-08-30 11:53:58 +02:00
Clément Renault	e6ffa4d454	Implement the document merge function for the replace method	2024-08-30 11:53:58 +02:00
Clément Renault	637a9c8bdd	Implement the document merge function for the update method	2024-08-30 11:53:58 +02:00
Louis Dureuil	c683fa98e6	WIP ... Co-authored-by: Kerollmops <clement@meilisearch.com> Co-authored-by: ManyTheFish <many@meilisearch.com>	2024-08-30 11:53:57 +02:00