Commit Graph

93 Commits

Author SHA1 Message Date
1818026a84 Merge #2057
2057: fix(dump): Uncompress the dump IN the data.ms  r=irevoire a=irevoire

When loading a dump with docker, we had two problems.
After creating a tempdirectory, uncompressing and re-indexing the dump:
1. We try to `move` the new “data.ms” onto the currently present
   one. The problem is that if the `data.ms` is a mount point because
   that's what peoples do with docker usually. We can't override
   a mount point, and thus we were throwing an error.
2. The tempdir is created in `/tmp`, which is usually quite small AND may not
   be on the same partition as the `data.ms`. This means when we tried to move
   the dump over the `data.ms`, it was also failing because we can't move data
   between two partitions.
------------------
1 was fixed by deleting the *content* of the `data.ms` and moving the *content*
of the tempdir *inside* the `data.ms`. If someone tries to create volumes inside
the `data.ms` that's his problem, not ours.
2 was fixed by creating the tempdir *inside* of the `data.ms`. If a user mounted
its `data.ms` on a large partition, there is no reason he could not load a big
dump because his `/tmp` was too small. This solves the issue; now the dump is
extracted and indexed on the same partition the `data.ms` will lay.

fix #1833

Co-authored-by: Tamo <tamo@meilisearch.com>
2022-01-10 17:57:16 +00:00
c9c7da3626 fix(dump): Uncompress the dump IN the data.ms
When loading a dump with docker, we had two problems.
After creating a tempdirectory, uncompressing and re-indexing the dump:
1. We try to `move` the new “data.ms” onto the currently present
   one. The problem is that if the `data.ms` is a mount point because
   that's what peoples do with docker usually. We can't override
   a mount point, and thus we were throwing an error.
2. The tempdir is created in `/tmp`, which is usually quite small AND may not
   be on the same partition as the `data.ms`. This means when we tried to move
   the dump over the `data.ms`, it was also failing because we can't move data
   between two partitions.
==============
1 was fixed by deleting the *content* of the `data.ms` and moving the *content*
of the tempdir *inside* the `data.ms`. If someone tries to create volumes inside
the `data.ms` that's his problem, not ours.
2 was fixed by creating the tempdir *inside* of the `data.ms`. If a user mounted
its `data.ms` on a large partition, there is no reason he could not load a big
dump because his `/tmp` was too small. This solves the issue; now the dump is
extracted and indexed on the same partition the `data.ms` will lay.

fix #1833
2022-01-10 14:56:03 +01:00
5fb4ed60e7 chore(all) set rust edition to 2021 2022-01-06 13:30:45 +01:00
b28a465304 bug(lib): drop env on last use
fixes the `too many open files` error when running tests by closing the
environment on last drop
2021-12-16 10:57:55 +01:00
ae73386723 Update version for the next release (v0.25.0) 2021-12-07 14:00:43 +01:00
23e35fa526 feat(dumps): drop dump V1 support 2021-12-07 10:36:27 +01:00
ae2b0e7aa7 Use milli reexported tokenizer instead of importing meilisearch-tokenizer dependency 2021-12-06 17:18:28 +01:00
948615537b Merge #1965
1965: Reintroduce engine version file r=MarinPostma a=irevoire

Right now if you boot up MeiliSearch and point it to a DB directory created with a previous version of MeiliSearch the existing indexes will be deleted. This [used to be](51d7c84e73) prevented by a startup check which would compare the current engine version vs what was stored in the DB directory's version file, but this functionality seems to have been lost after a few refactorings of the code.

In order to go back to the old behavior we'll need to reintroduce the `VERSION` file that used to be present; I considered reusing the `metadata.json` file used in the dumps feature, but this seemed like the simpler and more approach. As the intent is just to restore functionality, the implementation is quite basic. I imagine that in the future we could build on this and do things like compatibility across major/minor versions and even migrating between formats.

This PR was made thanks to `@mbStavola` and is basically a port of his PR #1860 after a big refacto of the code #1796.

Closes #1840

Co-authored-by: Matt Stavola <m.freitas@offensive-security.com>
2021-12-06 13:39:37 +00:00
a0e129304c feat(lib): Reintroduce engine version file
Right now if you boot up MeiliSearch and point it to a DB directory created with a previous version of MeiliSearch the existing indexes will be deleted. This used to be prevented by a startup check which would compare the current engine version vs what was stored in the DB directory's version file, but this functionality seems to have been lost after a few refactorings of the code.
In order to go back to the old behavior we'll need to reintroduce the VERSION file that used to be present; I considered reusing the metadata.json file used in the dumps feature, but this seemed like the simpler and more approach. As the intent is just to restore functionality, the implementation is quite basic. I imagine that in the future we could build on this and do things like compatibility across major/minor versions and even migrating between formats.

This PR was made thanks to @mbStavola

Closes #1840
2021-12-06 14:30:56 +01:00
ffefd0caf2 feat(auth): API keys
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0085-api-keys.md

- Add tests on API keys management route (meilisearch-http/tests/auth/api_keys.rs)
- Add tests checking authorizations on each meilisearch routes (meilisearch-http/tests/auth/authorization.rs)
- Implement API keys management routes (meilisearch-http/src/routes/api_key.rs)
- Create module to manage API keys and authorizations (meilisearch-auth)
- Reimplement GuardedData to extend authorizations (meilisearch-http/src/extractors/authentication/mod.rs)
- Change X-MEILI-API-KEY by Authorization Bearer (meilisearch-http/src/extractors/authentication/mod.rs)
- Change meilisearch routes to fit to the new authorization feature (meilisearch-http/src/routes/)

- close #1867
2021-12-06 09:52:41 +01:00
a30e02c18c feat(all): Task store
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0060-refashion-updates-apis.md

linked PR:

- #1889
- #1891
- #1892
- #1902
- #1906
- #1911
- #1914
- #1915
- #1916
- #1918
- #1924
- #1925
- #1926
- #1930
- #1936
- #1937
- #1942
- #1944
- #1945
- #1946
- #1947
- #1950
- #1951
- #1957
- #1959
- #1960
- #1961
- #1962
- #1964

- https://github.com/meilisearch/milli/pull/414
- https://github.com/meilisearch/milli/pull/409
- https://github.com/meilisearch/milli/pull/406
- https://github.com/meilisearch/milli/pull/418

- close #1687
- close #1786
- close #1940
- close #1948
- close #1949
- close #1932
- close #1956
2021-12-02 20:14:42 +01:00
c9f3726447 Merge #1893
1893: Make matches work with numerical value r=MarinPostma a=Thearas

# Pull Request

## What does this PR do?

Implement #1883.

I have test this PR with unit test. It appears to be working properly:
![image](https://user-images.githubusercontent.com/44015907/141141082-dad8cd18-e803-408f-ad6a-c7a212b7ec88.png)

PTAL `@curquiza` 

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?


Co-authored-by: Thearas <thearas850@gmail.com>
2021-11-29 13:38:57 +00:00
ac5535055f Make matches work with numerical value 2021-11-10 23:10:30 +08:00
cc6306c0e1 Update milli version 2021-11-04 14:57:45 +01:00
b664a46e91 Update milli version 2021-11-03 16:11:20 +01:00
06e6eaa7b4 Remove useless Facet variant 2021-11-03 16:11:09 +01:00
30a094cbb2 Change lacking errors 2021-11-03 14:33:33 +01:00
76a4f86e0c rename user-id to instance-uid 2021-10-29 17:25:52 +02:00
c752c14c46 refactorize the dump and snapshot 2021-10-29 17:25:51 +02:00
87a8bf5e96 write and load the user-id in the dumps 2021-10-29 17:25:51 +02:00
cf67964133 Merge #1848
1848: Error format and Definition r=MarinPostma a=ManyTheFish



Co-authored-by: many <maxime@meilisearch.com>
2021-10-28 14:15:35 +00:00
3a29cbf0ae Use milli v0.20.0 2021-10-28 15:59:06 +02:00
cbaca2b579 Fix PR comments 2021-10-28 15:42:42 +02:00
a76d9b15c9 Update version for the next release (v0.24.0) 2021-10-28 12:24:49 +02:00
59636fa688 Pimp error where no document is provided 2021-10-28 12:13:51 +02:00
7464720426 Fix some errors 2021-10-28 10:47:59 +02:00
d905bbf961 Merge #1787
1787: Handle empty dump r=MarinPostma a=irevoire

Fixes #1701

Co-authored-by: Tamo <tamo@meilisearch.com>
2021-10-27 12:47:45 +00:00
61c15b69fb Change malformed_payload error 2021-10-27 11:13:12 +02:00
4ac005b094 optimize document transform
fix error types

bump milli
2021-10-26 13:51:15 +02:00
5e3a53b576 fix a bug in the generation of empty dumps 2021-10-25 14:17:57 +02:00
36ab7b3ebd Fix small typo 2021-10-18 14:17:32 +02:00
b4038597ba Keep persisting tmp files in database directory and put non-persisting tmp files in default tmp dir 2021-10-18 14:16:35 +02:00
79817bd465 Merge #1813
1813: Apply highlight tags on numbers in the formatted search result output r=irevoire a=Jhnbrn90

This is my first ever Rust related PR. 

As described in #1758, I've attempted to highlighting numbers correctly under the `_formatted` key.

Additionally, I added a test which should assert appropriate highlighting. 

I'm open to suggestions and improvements. 


Co-authored-by: John Braun <john@brn.email>
2021-10-18 09:05:01 +00:00
be75426e64 Apply formatting according code style guidelines 2021-10-15 21:32:29 +02:00
0f342ac46e Update MeiliSearch version 2021-10-12 16:43:31 +02:00
29ac324e90 Update milli version to v0.17.3 2021-10-12 16:12:16 +02:00
3edbc74430 Merge branch 'main' into stable 2021-10-11 18:30:10 +02:00
60473637fe Update milli version 2021-10-11 16:21:19 +02:00
6c46fbbc57 Remove memmap dependency 2021-10-10 22:33:40 +01:00
3756f5a0ca Add test for highlighting numbers 2021-10-08 15:07:45 +02:00
5b4e4bb858 Highlight numbers (int) as string in formatted JSON 2021-10-08 15:07:15 +02:00
a92a0c3ed3 Log the error instead of returning it when deletion fails 2021-10-07 17:38:22 +02:00
0774b1efa5 Close index's heed environment when index is deleted 2021-10-07 17:09:41 +02:00
7fc7eb7457 Make sure to remove newly created index if uid is already taken 2021-10-07 16:49:21 +02:00
66dbd3cd34 makes clippy happy 2021-10-06 17:39:04 +02:00
ddbcf449da Merge #1763
1763: Index tests r=MarinPostma a=MarinPostma

This pr aims to test more thorougly the usage on index in the meilisearch database, by writing unit tests.

work included:
- [x] Create index mock and stub methods
- [x] Test snapshot creation
- [x] Test Dumps
- [x] Test search

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-10-06 14:39:53 +00:00
9fa61439b1 fix clippy warning & unsafety 2021-10-06 14:51:46 +02:00
a38215de98 edit documentation 2021-10-06 14:35:18 +02:00
85b5260d9d simple search unit test 2021-10-06 14:20:05 +02:00
4b4ebad9a9 test dumps 2021-10-06 14:10:26 +02:00