feat: [object-store] re-export hyper

[ritchie46]

Closes #5386

Are there any user-facing changes?

No

[tustvold]

You will need to add the license goop at the top to make CI happy

[ritchie46]

Ah, right.

[tustvold]

Is this a convention found in other crates, or do they re-export at the root?

[ritchie46]

Could not really find a convention. In this book it is done in the root: https://www.lurklurk.org/effective-rust/re-export.html

Which should work fine for hyper, but might give you some collisions if a library has a name colliding with internal modules. No preference on my side. Shall I do it from root?

[tustvold]

So I did some digging because reqwest, which is the client we are actually using, doesn't export hyper and I wondered why. I think it is something to do with the way that reqwest supports wasm32. This isn't an immediate issue, given wasm32 for HTTP stores is currently incomplete, but it does make me wonder if this is the correct way to go about this.

Perhaps something like #5345 might be what you are looking for?

[ritchie46]

Perhaps something like #5345 might be what you are looking for?

Which is an object-store variant of hyper/http errors? Seems also good. Since reqwest doesn't export hyper, but you do use it for error handling. Do you ensure manually to keep the versions matched?

[tustvold]

Do you ensure manually to keep the versions matched

Yes, and have tests that should catch any divergence. It is not an ideal situation, but it has worked so far

[ritchie46]

In that case, I would love to piggy back on that via an export. 😜

[tustvold]

Perhaps you might expand upon your use-case for downcasting the errors? It occurs to me that this is somewhat breaking encapsulation, and we might be able to expose this information in a better way, perhaps??

[ritchie46]

Perhaps you might expand upon your use-case for downcasting the errors? It occurs to me that this is somewhat breaking encapsulation, and we might be able to expose this information in a better way, perhaps??

I want to get most information out of the errors and see how to handle upon those. I haven't got a clear answer to what exactly my solution is. But having access to the http errors for accessing s3 is very important to us as I want to be close to maximum concurrency without triggering those. We want to be very fast here and like to be able experiment with tuning settings. Ideally I want to dynamically tune the client settings later.

It occurs to me that this is somewhat breaking encapsulation

Why? We can already get the hyper error out. It is only more painful now because we need to match the exact version. It is the same argument for your usage with reqwest. It would have been easier if they exported hyper. This gives power-users the ability to tinker.

[tustvold]

Why? We can already get the hyper error out. It is only more painful now because we need to match the exact version

Because there is no requirement that all or even any ObjectStore are implemented using hyper, and in fact we already have some that are not 😅. I accept that power users may want to tinker, but I guess I'm wondering if we really want to be encouraging this by making it a first-party API.

But having access to the http errors for accessing s3 is very important to us as I want to be close to maximum concurrency without triggering those.

Perhaps you could identify from the list of errors exposed by S3 what variant you are interested in intercepting. We automatically backoff on receiving a 503, which is what it will return if you exceed the maximum concurrency (unless you are really hammering it in which case it will just drop the connections).

[ritchie46]

Because there is no requirement that all or even any ObjectStore are implemented using hyper,

I totally understand that. And I as a user am willing to pay that breaking change risk. ;)

So we could export it under an unstable flag. Or if you don't want to expose it in object-store also fine, then we copy the hyper version from object-store and do it manually. Either way, I want to tinker. ^^

Perhaps you could identify from the list of errors exposed by S3 what variant you are interested in intercepting.

I don't know yet. I always learn while tinkering. 😅. It isn't a joke, I cannot answer what I need exactly yet. Have to learn this as we go.

If we have found something that works good for us, we can push that back into object-store. I rather have that handled higher up.

[tustvold]

In that case, how about we mark this PR as a draft, and revisit this once we have a better idea of the concrete use-cases for this feature?

[ritchie46]

Sure, fine by me. 👍