feat: Crypt driver, improve http/webdav handling (#4884)

this PR has several enhancements, fixes, and features:
- [x] Crypt: a transparent encryption driver. Anyone can easily, and safely store encrypted data on the remote storage provider.  Consider your data is safely stored in the safe, and the storage provider can only see the safe, but not your data.
  - [x] Optional: compatible with [Rclone Crypt](https://rclone.org/crypt/). More ways to manipulate the encrypted data.
  - [x] directory and filename encryption
  - [x] server-side encryption mode (server encrypts & decrypts all data, all data flows thru the server)
- [x] obfuscate sensitive information internally
- [x] introduced a server memory-cached multi-thread downloader.
  - [x] Driver: **Quark** enabled this feature, faster load in any single thread scenario. e.g. media player directly playing from the link, now it's faster.
- [x] general improvement on HTTP/WebDAV stream processing & header handling & response handling
  - [x] Driver: **Mega** driver support ranged http header
  - [x] Driver: **Quark** fix bug of not closing HTTP request to Quark server while user end has closed connection to alist

## Crypt, a transparent Encrypt/Decrypt Driver. (Rclone Crypt compatible)

e.g.  
Crypt mount path ->  /vault 
Crypt remote path -> /ali/encrypted
Aliyun mount paht -> /ali

when the user uploads a.jpg to /vault, the data will be encrypted and saved to /ali/encrypted/xxxxx. And when the user wants to access a.jpg,  it's automatically decrypted, and the user can do anything with it.
Since it's Rclone Crypt compatible, users can download /ali/encrypted/xxxxx  and decrypt it with rclone crypt tool. Or the user can mount this folder using rclone, then mount the decrypted folder in Linux...

NB.  Some breaking changes is made to make it follow global standard, e.g. processing the HTTP header properly.

close #4679 
close #4827 

Co-authored-by: Sean He <866155+seanhe26@users.noreply.github.com>
Co-authored-by: Andy Hsu <i@nn.ci>
This commit is contained in:
Sean
2023-08-02 14:40:36 +08:00
committed by GitHub
parent 1dc1dd1f07
commit 3c21a9a520
38 changed files with 2861 additions and 335 deletions

View File

@ -1,6 +1,7 @@
package model
import (
"github.com/alist-org/alist/v3/pkg/http_range"
"io"
"net/http"
"time"
@ -19,15 +20,16 @@ type LinkArgs struct {
}
type Link struct {
URL string `json:"url"`
Header http.Header `json:"header"` // needed header (for url) or response header(for data or writer)
Data io.ReadCloser // return file reader directly
Status int // status maybe 200 or 206, etc
FilePath *string // local file, return the filepath
Expiration *time.Duration // url expiration time
URL string `json:"url"`
Header http.Header `json:"header"` // needed header (for url) or response header(for data or writer)
RangeReadCloser RangeReadCloser // recommended way
ReadSeekCloser io.ReadSeekCloser // best for local,smb.. file system, which exposes ReadSeekCloser
Expiration *time.Duration // local cache expire Duration
IPCacheKey bool // add ip to cache key
//Handle func(w http.ResponseWriter, r *http.Request) error `json:"-"` // custom handler
Writer WriterFunc `json:"-"` // custom writer
//for accelerating request, use multi-thread downloading
Concurrency int
PartSize int
}
type OtherArgs struct {
@ -41,5 +43,10 @@ type FsOtherArgs struct {
Method string `json:"method" form:"method"`
Data interface{} `json:"data" form:"data"`
}
type RangeReadCloser struct {
RangeReader RangeReaderFunc
Closer io.Closer
}
type WriterFunc func(w io.Writer) error
type RangeReaderFunc func(httpRange http_range.Range) (io.ReadCloser, error)

View File

@ -21,6 +21,7 @@ type Obj interface {
GetName() string
ModTime() time.Time
IsDir() bool
//GetHash() (string, string)
// The internal information of the driver.
// If you want to use it, please understand what it means
@ -49,6 +50,9 @@ type Thumb interface {
type SetPath interface {
SetPath(path string)
}
type SetHash interface {
SetHash(hash string, hashType string)
}
func SortFiles(objs []Obj, orderBy, orderDirection string) {
if orderBy == "" {

View File

@ -29,6 +29,8 @@ type Object struct {
Size int64
Modified time.Time
IsFolder bool
Hash string
HashType string
}
func (o *Object) GetName() string {
@ -55,8 +57,17 @@ func (o *Object) GetPath() string {
return o.Path
}
func (o *Object) SetPath(id string) {
o.Path = id
func (o *Object) SetPath(path string) {
o.Path = path
}
func (o *Object) SetHash(hash string, hashType string) {
o.Hash = hash
o.HashType = hashType
}
func (o *Object) GetHash() (string, string) {
return o.Hash, o.HashType
}
type Thumbnail struct {