diff --git a/docs/embedder_settings.md b/docs/embedder_settings.md new file mode 100644 index 000000000..1098580e8 --- /dev/null +++ b/docs/embedder_settings.md @@ -0,0 +1,1398 @@ +The tables below have been generated by calling `cargo run --bin embedder_settings` + +## List of the embedder settings + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SettingDescriptionTypeDefault ValueRegenerate on Change
+ +`source` + + + + +The source used to provide the embeddings. + +Which embedder parameters are available and mandatory is determined by the value of this setting. + + + + +"openAi" | "huggingFace" | "userProvided" | "ollama" | "rest" | "composite" + + + +"openAi" + + + +🏗️ Always + +
+ +`model` + + + +The name of the model to use. + + + +string + + + + +- For source `openAi`, defaults to "text-embedding-3-small" +- For source `huggingFace`, defaults to "BAAI/bge-base-en-v1.5" + + + + +🏗️ Always + +
+ +`revision` + + + + +The revision (commit SHA1) of the model to use. + +If unspecified, Meilisearch picks the latest revision of the model. + + + + +string + + + + +- When `model` is set to default, defaults to "617ca489d9e86b49b8167676d8220688b99db36e" +- Otherwise, defaults to `null` + + + + +🏗️ Always + +
+ +`pooling` + + + +The pooling method to use. + + + +"useModel" | "forceCls" | "forceMean" + + + +"useModel" + + + +🏗️ Always + +
+ +`apiKey` + + + +The API key to pass to the remote embedder while making requests. + + + +string + + + +`null` + + + +🌱 Never + +
+ +`dimensions` + + + +The expected dimensions of the embeddings produced by this embedder. + + + +number + + + +`null` + + + + +- 🏗️ When the source is `openAi`, changing the value of this parameter always regenerates embeddings +- 🌱 For other sources, changing the value of this parameter never regenerates embeddings + + +
+ +`documentTemplate` + + + + +A liquid template used to render documents to a text that can be embedded. + +Meillisearch interpolates the template for each document and sends the resulting text to the embedder. +The embedder then generates document vectors based on this text. + + + + +string + + + +{% for field in fields %}{% if field.is_searchable and field.value != nil %}{{ field.name }}: {{ field.value }} +{% endif %}{% endfor %} + + + + +- 🏗️ When modified, embeddings are regenerated for documents whose rendering through the template produces a different text. + + +
+ +`documentTemplateMaxBytes` + + + +Rendered texts are truncated to this size before embedding. + + + +number + + + +400 + + + + +- 🏗️ When increased, embeddings are regenerated for documents whose rendering through the template produces a different text. +- 🌱 When decreased, embeddings are never regenerated + + +
+ +`url` + + + +URL to reach the remote embedder. + + + +string + + + +`null` + + + + +- 🌱 When modified for source `openAi`, embeddings are never regenerated +- 🏗️ When modified for sources `ollama` and `rest`, embeddings are always regenerated + + +
+ +`request` + + + +Template request to send to the remote embedder. + + + +any + + + +`null` + + + +🏗️ Always + +
+ +`response` + + + +Template response indicating how to find the embeddings in the response from the remote embedder. + + + +any + + + +`null` + + + +🏗️ Always + +
+ +`headers` + + + +Additional headers to send to the remote embedder. + + + +object + + + +`null` + + + +🌱 Never + +
+ +`searchEmbedder` + + + +Embedder settings for the embedder used at search time. + + + +object + + + +`null` + + + +🌱 Never + +
+ +`indexingEmbedder` + + + +Embedder settings for the embedder used at indexing time. + + + +object + + + +`null` + + + + +- Embedding are regenerated when the setting modified in the indexing embedder require regeneration. + + +
+ +`distribution` + + + +Affine transformation applied to the semantic score to make it more comparable to the ranking score. + + + +object + + + +`null` + + + +🌱 Never + +
+ +`binaryQuantized` + + + + +Whether to binary quantize the embeddings of this embedder. + +Binary quantized embeddings are smaller than regular embeddings, which improves +disk usage and retrieval speed, at the cost of relevancy. + + + + +boolean + + + +`false` + + + + +- Embeddings are not regenerated, but the binary quantization takes time during indexing. + + +
+ + +## Availability of the settings depending on the selected source + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Setting + +openAi + + + +huggingFace + + + +ollama + + + +userProvided + + + +rest + + + +composite + +
+ +`model` + + + +✅ Allowed + + + +✅ Allowed + + + +🔐 **Mandatory** + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + +
+ +`revision` + + + +🚫 Disallowed + + + +✅ Allowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + +
+ +`pooling` + + + +🚫 Disallowed + + + +✅ Allowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + +
+ +`apiKey` + + + +✅ Allowed + + + +🚫 Disallowed + + + +✅ Allowed + + + +🚫 Disallowed + + + +✅ Allowed + + + +🚫 Disallowed + +
+ +`dimensions` + + + +✅ Allowed + + + +🚫 Disallowed + + + +✅ Allowed + + + +🔐 **Mandatory** + + + +✅ Allowed + + + +🚫 Disallowed + +
+ +`documentTemplate` + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, ✅ Allowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, ✅ Allowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, ✅ Allowed + + + + +🚫 Disallowed + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, ✅ Allowed + + + + +🚫 Disallowed + +
+ +`documentTemplateMaxBytes` + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, ✅ Allowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, ✅ Allowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, ✅ Allowed + + + + +🚫 Disallowed + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, ✅ Allowed + + + + +🚫 Disallowed + +
+ +`url` + + + +✅ Allowed + + + +🚫 Disallowed + + + +✅ Allowed + + + +🚫 Disallowed + + + +🔐 **Mandatory** + + + +🚫 Disallowed + +
+ +`request` + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🔐 **Mandatory** + + + +🚫 Disallowed + +
+ +`response` + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🔐 **Mandatory** + + + +🚫 Disallowed + +
+ +`headers` + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +✅ Allowed + + + +🚫 Disallowed + +
+ +`searchEmbedder` + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🔐 **Mandatory** + +
+ +`indexingEmbedder` + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🚫 Disallowed + + + +🔐 **Mandatory** + +
+ +`distribution` + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + +
+ +`binaryQuantized` + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + + + + +- Usually, ✅ Allowed +- When used in `searchEmbedder` in a `composite` embedder, 🚫 Disallowed +- When used in `indexingEmbedder` in a `composite` embedder, 🚫 Disallowed + + +
+ \ No newline at end of file