storage: update azure blobstore client to use new sdk #5302
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MasslessParticle
requested review from
KMiller-Grafana and
a team
as code owners
MasslessParticle
commented
Feb 2, 2022
| @@ -288,7 +296,7 @@ replace github.com/sercand/kuberesolver => github.com/sercand/kuberesolver v2.4. | |||
|
|
|||
| replace github.com/hpcloud/tail => github.com/grafana/tail v0.0.0-20201004203643-7aa4e4a91f03 | |||
|
|
|||
| replace github.com/Azure/azure-sdk-for-go => github.com/Azure/azure-sdk-for-go v36.2.0+incompatible | |||
| replace github.com/Azure/azure-sdk-for-go/sdk/storage/azblob => github.com/MasslessParticle/azure-sdk-for-go/sdk/storage/azblob v0.2.1-0.20220131200443-9793e6f0cc65 | |||
There was a problem hiding this comment.
There is an open PR to msft from this fork that contains a deadlock fix. Point to it until the PR is merged
There was a problem hiding this comment.
MasslessParticle
commented
Feb 2, 2022
| @@ -37,23 +34,18 @@ const ( | |||
|
|
|||
| var ( | |||
| supportedEnvironments = []string{azureGlobal, azureChinaCloud, azureGermanCloud, azureUSGovernment} | |||
| noClientKey = azblob.ClientProvidedKeyOptions{} | |||
| endpoints = map[string]struct{ blobURLFmt, containerURLFmt string }{ | |||
| endpoints = map[string]struct{ blobURLFmt string }{ | |||
There was a problem hiding this comment.
sdk handles the container url now. We just have to care about which account we're signing into
MasslessParticle
commented
Feb 2, 2022
| @@ -80,77 +72,6 @@ var ( | |||
| } | |||
| ) | |||
|
|
|||
| // BlobStorageConfig defines the configurable flags that can be defined when using azure blob storage. | |||
| type BlobStorageConfig struct { | |||
There was a problem hiding this comment.
Moved this so the storage_client file only contains the client itself
MasslessParticle
commented
Feb 2, 2022
|
|
||
| // Process the blobs returned in this result segment (if the segment is empty, the loop body won't execute) | ||
| for _, blobInfo := range listBlob.Segment.BlobItems { | ||
| err := instrument.CollectedRequest(ctx, "azure.List", instrument.NewHistogramCollector(b.metrics.requestDuration), instrument.ErrorCode, func(ctx context.Context) error { |
There was a problem hiding this comment.
The new API means that this instrumentation is now for how long it takes to list all the blobs rather than each page of blobs
MasslessParticle
force-pushed
the
tpatterson/update-azure-blob-client
branch
from
edfb2a4
to
700f455
Compare
MasslessParticle
force-pushed
the
tpatterson/update-azure-blob-client
branch
from
700f455
to
b0c4098
Compare
owen-d
approved these changes
Feb 2, 2022
KMiller-Grafana
pushed a commit
to KMiller-Grafana/loki
that referenced
this pull request
Feb 4, 2022
* Update azure blobstore client to use new sdk * changelog * fix hardcoded maxretries * linter * more linting * Fix flaky test
MasslessParticle
added a commit
to MasslessParticle/loki
that referenced
this pull request
Feb 16, 2022
owen-d
pushed a commit
that referenced
this pull request
Feb 16, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates the Azure Blobstore client to use the new
azure-sdk-for-go/sdk/storagepackage. It basically amounts to a rewrite.The huge number of files changed is because of vendored dependencies.
The files to pay attention to are all in the
storage/chunk/azurepackage. I also took this as an opportunity to refactor the Azure metrics and config to their own files soblob_storage_client.gois where the meat of this PR liesChecklist
CHANGELOG.mdabout the changes.