Large File Upload Support

[t83714]

Large File Upload Support

Magda allow you to store the data file into either internal storage (k8s PV) or cloud storage via our storage API (backed by minio).

However, the current web UI might not work for a file in very large size (e.g. 10GB) as:

• Our UI read whole file content in one go
• The storage API process the whole file in one request

We should:

• As part of Reshape Storage API #3335, upload file via resigned URL only
• Process file in chunks at the frontend (using file slice API) and upload to storage via AWS multi-part protocol (supported by minio)
• Our implementation should also support resuming a previously dropped off upload
  • The current create dataset UI already has the "recover previous change" function. We should extend this function to cover the file upload process as well

Acceptance Criteria

• The feature can be turned on / off via web-server Helm Chart config multipartUpload.enabled
• the part size can be config via multipartUpload.partSize with default value 6MB. If a file is smaller than the part size, we will not upload the file with multipart mode.
• If a multipart upload process is interrupted (e.g. due to network error or browser crash etc.), we should allow user to resume the upload without re-upload those complete parts.
• As "List Multipart Uploads always returns an empty list" when using GCS & Azure blob, we should find a way to accommodate the cloud vendor implementation differences and make sure the function works consistently across AWS S3, GCS & Azure blob storage options.