Per-bucket configuration

[dsuess]

Fsspec allows you to set global settings through their configuration mechanisms. However, things like endpoint_url can actually depend on the bucket you're trying to access.

Is there any existing way of configuring certain settings in s3fs in a bucket-dependent way? If not, would this be feasible to implement in the current architecture?

[martindurant]

No, there isn't an obvious way to do per-bucket configuration, although of course you can create as many filesystem instances as you like.

Does the argument cache_regions do anything useful for you?

[Erotemic]

I'm also looking for this features. I have two buckets s3://bucket1 and s3://bucket2. One of them requires requester_pays=True and the other requires requester_pays=False. I was trying to setup a registry, pre-register filesystems that map a bucket to a specific configuration, and then have paths mapped to the correct filesystem. The proof of concept works, but I don't know how to do it in a general way that works with other backends like ssh.

[martindurant]

I could imagine maybe the functions that work on whole URLs, like fsspec.open() checking for configuration beyond the top-level one specified by the protocol ("s3"); but once you have a configured instance, it cannot change itself, so s3.open will use the instance configuration regardless of the bucket.