402: Optimize document transform r=MarinPostma a=MarinPostma

This pr optimizes the transform of documents additions in the obkv format. Instead on accepting any serializable objects, we instead treat json and CSV specifically:
- For json, we build a serde `Visitor`, that transform the json straight into obkv without intermediate representation.
- For csv, we directly write the lines in the obkv, applying other optimization as well.

Co-authored-by: marin postma <postma.marin@protonmail.com>
This commit is contained in:
bors[bot]
2021-10-26 09:55:28 +00:00
committed by GitHub
14 changed files with 706 additions and 818 deletions

View File

@ -48,6 +48,7 @@ itertools = "0.10.0"
# logging
log = "0.4.14"
logging_timer = "1.0.0"
csv = "1.1.6"
[dev-dependencies]
big_s = "1.0.2"