Commit Graph

93 Commits

Author SHA1 Message Date
457a473b72 Bring back release-v0.30.0 into release-v0.30.0-temp (final: into main) (#3145)
* Fix error code of the "duplicate index found" error

* Use the content of the ProcessingTasks in the tasks cancelation system

* Change the missing_filters error code into missing_task_filters

* WIP Introduce the invalid_task_uid error code

* Use more precise error codes/message for the task routes

+ Allow star operator in delete/cancel tasks
+ rename originalQuery to originalFilters
+ Display error/canceled_by in task view even when they are = null
+ Rename task filter fields by using their plural forms
+ Prepare an error code for canceledBy filter
+ Only return global tasks if the API key action `index.*` is there

* Add canceledBy task filter

* Update tests following task API changes

* Rename original_query to original_filters everywhere

* Update more insta-snap tests

* Make clippy happy

They're a happy clip now.

* Make rustfmt happy

>:-(

* Fix Index name parsing error message to fit the specification

* Bump milli version to 0.35.1

* Fix the new error messages

* fix the error messages and add tests

* rename the error codes for the sake of consistency

* refactor the way we send the cli informations + add the analytics for the config file and ssl usage

* Apply suggestions from code review

Co-authored-by: Clément Renault <clement@meilisearch.com>

* add a comment over the new infos structure

* reformat, sorry @kero

* Store analytics for the documents deletions

* Add analytics on all the settings

* Spawn threads with names

* Spawn rayon threads with names

* update the distinct attributes to the spec update

* update the analytics on the search route

* implements the analytics on the health and version routes

* Fix task details serialization

* Add the question mark to the task deletion query filter

* Add the question mark to the task cancelation query filter

* Fix tests

* add analytics on the task route

* Add all the missing fields of the new task query type
* Create a new analytics for the task deletion
* Create a new analytics for the task creation

* batch the tasks seen events

* Update the finite pagination analytics

* add the analytics of the swap-indexes route

* Stop removing the DB when failing to read it

* Rename originalFilters into originalFilters

* Rename matchedDocuments into providedIds

* Add `workflow_dispatch` to flaky.yml

* Bump grenad to 0.4.4

* Bump milli to version v0.37.0

* Don't multiply total memory returned by sysinfo anymore

sysinfo now returns bytes rather than KB

* Add a dispatch to the publish binaries workflow

* Fix publish release CI

* Don't use gold but the default linker

* Always display details for the indexDeletion task

* Fix the insta tests

* refactorize the whole test suite
1. Make a call to assert_internally_consistent automatically when snapshoting the scheduler. There is no point in snapshoting something broken and expect the dumb humans to notice.
2. Replace every possible call to assert_internally_consistent by a snapshot of the scheduler. It takes as many lines and ensure we never change something without noticing in any tests ever.
3. Name every snapshots: it's easier to debug when something goes wrong and easier to review in general.
4. Stop skipping breakpoints, it's too easy to miss something. Now you must explicitely show which path is the scheduler supposed to use.
5. Add a timeout on the channel.recv, it eases the process of writing tests, now when something file you get a failure instead of a deadlock.

* rebase on release-v0.30

* makes clippy happy

* update the snapshots after a rebase

* try to remove the flakyness of the failing test

* Add more analytics on the ranking rules positions

* Update the dump test to check for the dumpUid dumpCreation task details

* send the ranking rules as a string because amplitude is too dumb to process an array as a single value

* Display a null dumpUid until we computed the dump itself on disk

* Update tests

* Check if the master key is missing before returning an error

Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2022-11-28 16:27:41 +01:00
2f577b6fcd Patch the IndexScheduler in meilisearch-http to use the options struct 2022-10-27 11:35:16 +02:00
4736e00253 Handle the CLI options related to snapshots 2022-10-27 11:35:15 +02:00
942b7c338b Compress the snapshot in a tarball 2022-10-27 11:35:15 +02:00
4cafc63561 Reintroduce the versioning functions 2022-10-27 11:35:14 +02:00
89e127e4f4 Declare the auth path in the index scheduler 2022-10-27 11:35:14 +02:00
c063f154fb Add the snapshots directory path to the IndexScheduler 2022-10-27 11:35:14 +02:00
8a23e707c1 fix the task view and forward the task db size 2022-10-27 11:35:09 +02:00
e9055f5572 fix clippy 2022-10-27 11:35:08 +02:00
ca4234b445 fix the deletion of the data.ms in case of failure 2022-10-27 11:35:06 +02:00
80b2e70ee7 Introduce a rustfmt file 2022-10-27 11:35:05 +02:00
725158b454 Introduce the core algorithm of task cancelation 2022-10-27 11:34:11 +02:00
634eb52926 extract the create_app function for the tests 2022-10-27 11:34:09 +02:00
6bd6321226 dump the content of the dump tasks instead of recreating at import time with wrong API keys 2022-10-27 11:34:08 +02:00
9fe24fbff2 get rids of the useless Seek before creating a grenad reader 2022-10-27 11:34:08 +02:00
78ce29f461 apply most style comments of the review 2022-10-27 11:34:08 +02:00
e0221fc0a3 fix a synchronization bug while importing tasks 2022-10-27 11:34:07 +02:00
554600dfd8 fix the deletion of the data.ms in case of errors 2022-10-27 11:34:07 +02:00
d976e680c5 first mostly working version 2022-10-27 11:34:06 +02:00
b7f9c94f4a write the dump export 2022-10-27 11:34:05 +02:00
667c282e19 get rids of the index crate + the document_types crate 2022-10-27 11:34:00 +02:00
91e13c2824 Implement ErrorCode on the milli::Error type 2022-10-27 11:33:48 +02:00
fba9aa214a remove the create_app macro 2022-10-27 11:33:45 +02:00
2c8f1a43e9 get rids of meilisearch-lib 2022-10-27 11:33:44 +02:00
8d51c1f389 wip integrating the scheduler in meilisearch-http 2022-10-27 11:33:42 +02:00
6aa3ad6b6c move prometheus behind a feature flag 2022-08-29 14:36:59 +02:00
2f92169e48 clippy issue in metrics fixed 2022-08-22 13:21:22 +05:30
2b8f3c26ec Changed prometheus metrics feature as optional 2022-08-22 13:21:22 +05:30
25fecf9360 clippy & rustfmt fixed 2022-08-22 13:21:22 +05:30
4bee0565e8 prometheus and grafana dashboards implemented 2022-08-22 13:21:22 +05:30
ae174c2cca Fix task serialization 2022-08-11 13:35:35 +02:00
e6f03f82df Fix clippy warnings 2022-07-28 15:56:22 +02:00
58d2aad309 Change binary option and add env var support 2022-07-28 15:13:49 +02:00
e3426d5b7a Improve the tasks error reporting 2022-07-28 15:12:54 +02:00
aff8cd1774 Make clippy happy 2022-07-04 13:36:56 +02:00
36cb09eb25 Add a new meilisearch_types crate
Move `meilisearch_error` to `meilisearch_types::error`
Move `meilisearch_lib::index_resolver::IndexUid` to `meilisearch_types::index_uid`
Add a new `InvalidIndexUid` error in `meilisearch_types::index_uid`
2022-06-09 16:14:13 +02:00
3684c822f1 Add indexUid filtering on the /tasks route 2022-05-31 11:33:20 +02:00
62ce8e0bda chore(http): rename auto batching cli option 2022-03-07 15:19:19 +01:00
3b2e467ca6 update actix-web dependency to 4.0 2022-02-28 19:03:37 +01:00
c9a236b0af feat(lib): auto-batching 2022-02-01 18:06:20 +01:00
c8bb54cd94 Merge #2098
2098: feat(dump): Provide the same cli options as the snapshots r=MarinPostma a=irevoire

Add two cli options for the dump:
- `--ignore-missing-dump`
- `--ignore-dump-if-db-exists`

Fix #2087

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-01-26 14:32:23 +00:00
bad4bed439 feat(dump): Provide the same cli options as the snapshots
Add two cli options for the dump:
- `--ignore-missing-dump`
- `--ignore-dump-if-db-exists`

Fix #2087
2022-01-26 14:34:06 +01:00
9f5fee404b chore(all): update actix-web dependency to 4.0.0-beta.21 2022-01-21 20:44:17 +00:00
595250c93e Allow any header for CORS 2022-01-05 15:38:47 +01:00
450b81ca13 Bug(CORS): Add missing allowed headers
fix #2040
2022-01-03 13:41:12 +01:00
ee7970f603 feat(auth): Extend API keys
- Add API keys in snapshots
- Add API keys in dumps
- Rename action indexes.add to indexes.create
- fix QA #1979

fix #1979
fix #1995
fix #2001
fix #2003
related to #1890
2021-12-14 17:33:39 +01:00
ffefd0caf2 feat(auth): API keys
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0085-api-keys.md

- Add tests on API keys management route (meilisearch-http/tests/auth/api_keys.rs)
- Add tests checking authorizations on each meilisearch routes (meilisearch-http/tests/auth/authorization.rs)
- Implement API keys management routes (meilisearch-http/src/routes/api_key.rs)
- Create module to manage API keys and authorizations (meilisearch-auth)
- Reimplement GuardedData to extend authorizations (meilisearch-http/src/extractors/authentication/mod.rs)
- Change X-MEILI-API-KEY by Authorization Bearer (meilisearch-http/src/extractors/authentication/mod.rs)
- Change meilisearch routes to fit to the new authorization feature (meilisearch-http/src/routes/)

- close #1867
2021-12-06 09:52:41 +01:00
a30e02c18c feat(all): Task store
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0060-refashion-updates-apis.md

linked PR:

- #1889
- #1891
- #1892
- #1902
- #1906
- #1911
- #1914
- #1915
- #1916
- #1918
- #1924
- #1925
- #1926
- #1930
- #1936
- #1937
- #1942
- #1944
- #1945
- #1946
- #1947
- #1950
- #1951
- #1957
- #1959
- #1960
- #1961
- #1962
- #1964

- https://github.com/meilisearch/milli/pull/414
- https://github.com/meilisearch/milli/pull/409
- https://github.com/meilisearch/milli/pull/406
- https://github.com/meilisearch/milli/pull/418

- close #1687
- close #1786
- close #1940
- close #1948
- close #1949
- close #1932
- close #1956
2021-12-02 20:14:42 +01:00
d65f055030 pass anaytics into Arc instead of static ref 2021-10-29 17:25:55 +02:00
d72c887422 makes the analytics available for all the routes 2021-10-29 17:25:47 +02:00