Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update code to fix almost all pre-existing lint errors #2008

Merged
merged 31 commits into from Feb 24, 2020

Conversation

zendern
Copy link
Contributor

@zendern zendern commented Jan 20, 2020

What this PR does:
Cleanups all but one of the lint errors that were found when the hash was in place.

Last outstanding issue is as follows

pkg/querier/frontend/worker.go:60:10: SA1019: naming.Watcher is deprecated: please use package resolver.  (staticcheck)
	watcher naming.Watcher

Not 100% clear yet on path forward on this. From what i can tell even the underlying library is still using naming.Watcher. Wondering if for now we put a //nolint:staticcheck and open an issue to fix just that issue.

Which issue(s) this PR fixes:
Fixes #1912

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@zendern
Copy link
Contributor Author

zendern commented Jan 21, 2020

@jtlisi let me know if you have feeling or thoughts on how to handle the last outstanding lint issue. Tagged you since you were the original issue reporter.

Copy link
Contributor

@jtlisi jtlisi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a minor nit. Also for that final linting error, feel free to add //nolint:staticcheck and open an issue.

Thanks for taking this on

pkg/chunk/gcp/bigtable_index_client.go Show resolved Hide resolved
pkg/util/grpcclient/backoff_retry.go Outdated Show resolved Hide resolved
@zendern zendern force-pushed the update-lint branch 2 times, most recently from d7848a8 to 8b6ce8e Compare January 21, 2020 18:32
@zendern
Copy link
Contributor Author

zendern commented Jan 21, 2020

Opened #2015 to deal with that last outstanding lint issue. All checks are now passing. @jtlisi should be ready for you to take a look at again.

Copy link
Contributor

@bboreham bboreham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! I made a lot of comments; I think the FramedSnappy one is a real issue.

pkg/chunk/aws/dynamodb_storage_client_test.go Outdated Show resolved Hide resolved
@@ -75,7 +75,7 @@ func (i *instrumentedCache) Store(ctx context.Context, keys []string, bufs [][]b
}

method := i.name + ".store"
instr.CollectedRequest(ctx, method, requestDuration, instr.ErrorCode, func(ctx context.Context) error {
_ = instr.CollectedRequest(ctx, method, requestDuration, instr.ErrorCode, func(ctx context.Context) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I question why Store() has no error return.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW from pkg/chunk/cache/cache.go

// Cache byte arrays by key.
//
// NB we intentionally do not return errors in this interface - caching is best
// effort by definition.  We found that when these methods did return errors,
// the caller would just log them - so its easier for implementation to do that.
// Whats more, we found partially successful Fetchs were often treated as failed
// when they returned an error.
type Cache interface {
	Store(ctx context.Context, key []string, buf [][]byte)
	Fetch(ctx context.Context, keys []string) (found []string, bufs [][]byte, missing []string)
	Stop() error
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bboreham just double checking are we good with this?? as per the comment above it was very intentional.

pkg/chunk/chunk_store.go Outdated Show resolved Hide resolved
pkg/chunk/chunk_store_test.go Show resolved Hide resolved
pkg/chunk/chunk_store_utils.go Outdated Show resolved Hide resolved
pkg/util/grpcclient/ratelimit.go Outdated Show resolved Hide resolved
pkg/util/http.go Outdated Show resolved Hide resolved
pkg/chunk/aws/s3_storage_client.go Outdated Show resolved Hide resolved
pkg/chunk/cache/memcached.go Show resolved Hide resolved
pkg/distributor/billing.go Show resolved Hide resolved
Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job @zendern. Looking at the change set, and the possible issues spotted, it's easy to understand the benefits of the linter. I've also left some comments.

pkg/chunk/aws/dynamodb_storage_client.go Show resolved Hide resolved
pkg/chunk/aws/dynamodb_storage_client.go Show resolved Hide resolved
pkg/chunk/cache/memcached.go Outdated Show resolved Hide resolved
pkg/chunk/schema_config.go Outdated Show resolved Hide resolved
pkg/querier/frontend/frontend.go Outdated Show resolved Hide resolved
pkg/querier/queryrange/instrumentation.go Show resolved Hide resolved
@zendern zendern force-pushed the update-lint branch 2 times, most recently from fcc1c31 to de4c97e Compare January 28, 2020 04:20
limiter.Wait(ctx)
err := limiter.Wait(ctx)
if err != nil {
return err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When used in conjunction with NewBackoffRetry() (see grpcclient.go), the backoff expects a ResourceExhausted error code.

@bboreham what do you think returning that GRPC error here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like a bug.
"Error: I'm not going to wait because the context will expire before the wait finishes"
"Never mind, try again!"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So @bboreham / @pracucci if i am understanding correctly what you are saying is that making this change is the wrong thing to do. Basically since the wait would not return a ResourceExhausted that would cause the BackoffRetry, if its being used, to now exit and no longer retry like we expect it to.

So if that is correct the right course of action is to just go back to what we had in originally here
#2008 (comment)
and ignore the error or should the BackoffRetry code check for another error code along with ResourceExhausted.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not rollback but fix it. We may do:
return status.New(codes.ResourceExhausted, err.Error())

But I would be glad to also see a unit test on it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll look at adding a unit test for this and getting that change in place.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok thanks! This looks like the very last thing left to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pracucci So i wrote up this unit test last night.....which might not be enough honestly. It's fine that it can validate that its a status with the expected result but an integration test would probably drive home the fact that it's necessary if you configure it with the backoffRetry. I just ran out of time.....I'll see if i can get to it over the weekend.


import (
	"context"
	"testing"

	"github.com/cortexproject/cortex/pkg/util/grpcclient"
	assert2 "github.com/stretchr/testify/assert"
	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"
)

func TestRateLimiterFailureResultsInResourceExhaustedError(t *testing.T) {
	config := grpcclient.Config{
		RateLimitBurst: 0,
		RateLimit: 0,
	}
	conn := grpc.ClientConn{}
	invoker := func(currentCtx context.Context, currentMethod string, currentReq, currentRepl interface{}, currentConn *grpc.ClientConn, currentOpts ...grpc.CallOption) error {
		return nil
	}

	limiter := grpcclient.NewRateLimiter(&config)
	err := limiter(context.Background(), "methodName", "", "expectedReply", &conn, invoker)
	
	if se, ok := err.(interface {
		GRPCStatus() *status.Status
	}); ok {
		assert2.Equal(t, se.GRPCStatus().Code(), codes.ResourceExhausted)
		assert2.Equal(t, se.GRPCStatus().Message(), "rate: Wait(n=1) exceeds limiter's burst 0")
	} else{
		assert2.Fail(t, "Could not convert error into expected Status type")
	}
}

pkg/storage/tsdb/config.go Outdated Show resolved Hide resolved
pkg/ring/lifecycler.go Show resolved Hide resolved
pkg/querier/frontend/frontend.go Outdated Show resolved Hide resolved
pkg/distributor/billing.go Show resolved Hide resolved
pkg/chunk/cache/memcached.go Show resolved Hide resolved
pkg/chunk/aws/s3_storage_client.go Outdated Show resolved Hide resolved
@zendern
Copy link
Contributor Author

zendern commented Jan 28, 2020

@bboreham @pracucci so looking into golangci and errcheck i dont believe I am able to add the excludes on (hash.Hash).Write as of now. golangci has forked errcheck and is using their own version.

Using errcheck directly the above exclude works b/c they have built in support for embedded interfaces. Whereas the golangci fork of errcheck is out of date and does not have that code.

https://github.com/golangci/errcheck/tree/master/internal/errcheck

https://github.com/kisielk/errcheck/tree/master/internal/errcheck

I'm working on a PR to try to bring the golangci-lint version up to the same as errcheck master but that will take a little time. So how would you like to proceed??

@pracucci
Copy link
Contributor

so looking into golangci and errcheck i dont believe I am able to add the excludes on (hash.Hash).Write as of now.
So how would you like to proceed??

Personally I'm fine even without an override if it's not easy to do. All in all, it affect 3 places and we can always re-iterate on it once supported.

Copy link
Contributor

@jtlisi jtlisi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the scope is limited, I'm fine without the override as well

@zendern zendern force-pushed the update-lint branch 2 times, most recently from 6c3f5ad to 741e440 Compare February 13, 2020 15:26
if err != nil {
panic(err)
}
return chunk
}

func TeardownFixture(t *testing.T, fixture Fixture) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any specific reason why this function was introduced? It looks a bit superfluous to me, given we can directly call require.NoError(t, fixture.Teardown()) instead of TeardownFixture().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was done in this commit. 8bf4365 It was just something i noticed we were doing 2 places so felt like a natural thing since the fixture was already in the testutils.go class. I could go either way....ill leave that up to you but that was the reasoning.

limiter.Wait(ctx)
err := limiter.Wait(ctx)
if err != nil {
return err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not rollback but fix it. We may do:
return status.New(codes.ResourceExhausted, err.Error())

But I would be glad to also see a unit test on it.

@pracucci
Copy link
Contributor

@zendern Could you attempt another rebase and address the open comment, please? I'm hearing your pain continuously rebasing this: I'm going to stress other maintainers to review this as well, so that we can get this merged.

@zendern zendern force-pushed the update-lint branch 3 times, most recently from 87f183c to ea7b3aa Compare February 24, 2020 02:59
To be specific ineffassign, deadcode and unused errors

Signed-off-by: Nathan Zender <github@nathanzender.com>
keys using hashing

Signed-off-by: Nathan Zender <github@nathanzender.com>
Now that we have an iterator there is no need to also have a consumed
bool on the underlying object.

Signed-off-by: Nathan Zender <github@nathanzender.com>
Will never actually be used. Only necessary to pad out CPU cache lines.

Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Attempted not to change any existing logic so if an error was ignored we
will now either explicitly ignore it or in the case of a goroutine being
called with a func that is ignoring the error we will just put a
//noling:errcheck on that line.

If it was in a test case we went ahead and did the extra assertion
checks b/c its always good to know where something might have errored in
your test cases.

Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
We have opened up issue
cortexproject#2015 to address the
deprecation of this type.

Signed-off-by: Nathan Zender <github@nathanzender.com>
The test is currently failing due to a data race. I believe it is due to
this bug in golang.

golang/go#30597

As far as this test cares it does not really matter that this happens so
removing the need to check for NoError "fixes" it.

Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Since Fixture is already in testutils and it is being used in both
places pulled it out into a common helper method in the testutils
package.

Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Signed-off-by: Nathan Zender <github@nathanzender.com>
Formatting and import issues that got missed when merging.

Signed-off-by: Nathan Zender <github@nathanzender.com>
This is necessary so that when it is used with the backoff retry it will
allow for the backoff to continue to work as expected.

Signed-off-by: Nathan Zender <github@nathanzender.com>
Copy link
Contributor

@bboreham bboreham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work, thanks!
Let's merge it quick before anyone else blinks.

Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, awesome work! Let's merge this! 🎉

@pracucci pracucci merged commit 822d699 into cortexproject:master Feb 24, 2020
@pracucci pracucci mentioned this pull request Feb 24, 2020
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pre Golang-CI Linting Errors
4 participants