Metric bucket should not return error when error is expected

[yeya24]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Now, when metric bucket got an error which is isOpFailureExpected, it doesn't increment the metric but still returns the error back to the caller.

For example, it is quite common to log the error returned by bucket operations like below. However, even if I already specify IsXXXError as expected, the error still returns and will be logged. To filter it out, I need to add another IsXXXError(err) check to not log the error, which is very redundant to me.

err := userBucket.ReaderWithExpectedErrs(IsXXXError).Iter(...)
level.Log("msg", "got error", "err", err)

We also have another usecase which doesn't support checking the error in the caller. https://github.com/cortexproject/cortex/blob/master/pkg/storage/bucket/s3/bucket_client.go#L135

We implemented our own storage provider which has retries. Since the error is returned, there is no way for the retryer to know if the error is expected or not and it will keep retrying and finally log the error. There is no way to inject the expected error to the bucket client itself in this case.

Verification