Memory usage

[ghost]

why does it take 14GB+ usage of ram to upload a 5GB file?

[jkawamoto]

The library this app uses needs to copy the uploading file into RAM at least two times to build a request. That means it'll consume at least 10GB RAM to upload a 5GB file.

Btw, can you upload a 5GB file? The API document looks like the max file size is 1GB:

[ghost]

yes, you can upload 5GB via API. 15GB if you have pro account, etc.

i emailed pixeldrain support and Wim gave me a go app for uploading: https://pixeldrain.com/u/V8UGqPb4
pd "file/location/here"

does not consume ram which is an issue with your app i have as I upload large file simultaneously and i run out of ram.

Wim's app does not copy files into ram it seems, no usage at all. bottleneck is going to be your upload speed to PD not disks so i don't see need to copy to ram. maybe you can rework that. =)

btw if you email PD suport wim will provide source code for the go app i linked above if you are interested.

[jkawamoto]

I see. I've just opened a PR go-openapi/runtime#214. It'll reduce the RAM usage.

[ghost]

I see, thank you :)

[Fornax96]

Here's the source code for my uploader: https://pixeldrain.com/u/VmcG7r4f

It spawns a separate goroutine to create the multipart file in the background.

[jkawamoto]

Thanks for the source code. It looks like go-openapi/runtime, which is used in this app, also uses a goroutine to prepare the multipart message:

https://github.com/go-openapi/runtime/blob/2bc043c8ba8a760fd663163517bac1fb69fd717a/client/request.go#L131-L186

It wan't optimal to compute a content type but it's fixed now.