Add transfer configuration to support concurrent downloading

[ser-amzn]

I am trying to speed up downloading of large files from S3.

I can enable parallel downloading of file parts in boto3 using https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3.html:

    config = TransferConfig(
        multipart_threshold=1000000000,
        multipart_chunksize=50000000,
        max_concurrency=128,
    )
    s3_client.download_file(bucket, key, dst, Config=config)

Can I do the same in s3fs? if seems that passing s3_additional_kwargs doesn't help:

s3 = s3fs.S3FileSystem(anon=False, s3_additional_kwargs={'Config': config}) 

It would be great to have this feature in s3fs.

[martindurant]

I don't believe that requesting parallel streams for a single file will increase bandwidth - you should simply saturate your network bandwidth once the transfer begins. Concurrency helps most when making many requests and wishing to amortise the latency.

(decoding and compression may parallelise well, but they are not normally significant, and python's GIL makes it hard to achieve)

Having said that, s3transfer does do some clever things, so if anyone is interested in calling it within get/download[_file], I would be interested to see this.