Azure Blob: Unable to access Metadata when listing blobs #16679

jkowalski · 2021-12-17T01:46:50Z

Bug Report

import path of package in question, e.g. github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

require github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.2.0

require (
github.com/Azure/azure-sdk-for-go/sdk/azcore v0.20.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v0.8.1 // indirect
github.com/stretchr/objx v0.2.0 // indirect
)

output of go version
go version go1.17.2 darwin/arm64
What happened?

I'm trying to list metadata associated with blobs, but it appears to be incorrectly handled in the SDK code so it always comes back empty.

To reproduce, upload some blobs with custom metadata like so:

	resp, err := bc.Upload(ctx, data.Reader(), &azblob.UploadBlockBlobOptions{
		Metadata: map[string]string{"Kopiamtime": "1577968240000000000"},
	})

Now when listing the blobs there are 2 sub-issues, one blocking the other:

Incorrect handlng of ContainerListBlobFlatSegmentOptions.Include:

My first attempt was to try:

bucket.ListBlobsFlat(&azblob.ContainerListBlobFlatSegmentOptions{
		Prefix: &prefixStr,
		Include: []azblob.ListBlobsIncludeItem{azblob.ListBlobsIncludeItemMetadata},
	})

but the server never returns any metadata. I read through the code and I think there's a bug generating the include query parameter and server silently ignores invalid value (bug in itself?), and to overcome it I have to add artificial square brackets, which makes it work partially - see below:

bucket.ListBlobsFlat(&azblob.ContainerListBlobFlatSegmentOptions{
		Prefix: &prefixStr,
		Include: []azblob.ListBlobsIncludeItem{"["+azblob.ListBlobsIncludeItemMetadata+"]"},
	})

Incorrect mapping of Metadata

After applying the workaround above, I get the following response from the server (there are 4 files in the bucket that match the prefix) - the response was captured using io.Copy(os.Stderr, resp.RawResponse.Body):

<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults ServiceEndpoint="https://kopiatesting.blob.core.windows.net/" ContainerName="kopia-testing">
    <Prefix>sastest-1639703094-ed1304df8b99da34-</Prefix>
    <Blobs>
        <Blob>
            <Name>sastest-1639703094-ed1304df8b99da34-abcdbbf4f0507d054ed5a80a5b65086f602b</Name>
            <Properties>
                <Creation-Time>Fri, 17 Dec 2021 01:04:55 GMT</Creation-Time>
                <Last-Modified>Fri, 17 Dec 2021 01:04:55 GMT</Last-Modified>
                <Etag>0x8D9C0F93CEC42AE</Etag>
                <Content-Length>0</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-CRC64 />
                <Content-MD5>1B2M2Y8AsgTpgAmY7PhCfg==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <AccessTier>Hot</AccessTier>
                <AccessTierInferred>true</AccessTierInferred>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>true</ServerEncrypted>
            </Properties>
            <Metadata>
                <Kopiamtime>1577968240000000000</Kopiamtime>
            </Metadata>
            <OrMetadata />
        </Blob>
        <Blob>
            <Name>sastest-1639703094-ed1304df8b99da34-abff4585856ebf0748fd989e1dd623a8963d</Name>
            <Properties>
                <Creation-Time>Fri, 17 Dec 2021 01:04:55 GMT</Creation-Time>
                <Last-Modified>Fri, 17 Dec 2021 01:04:55 GMT</Last-Modified>
                <Etag>0x8D9C0F93D28CE3A</Etag>
                <Content-Length>1000</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-CRC64 />
                <Content-MD5>8INdvman6QsODoZHNU0MCw==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <AccessTier>Hot</AccessTier>
                <AccessTierInferred>true</AccessTierInferred>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>true</ServerEncrypted>
            </Properties>
            <Metadata>
                <Kopiamtime>1577968240000000000</Kopiamtime>
            </Metadata>
            <OrMetadata />
        </Blob>
        <Blob>
            <Name>sastest-1639703094-ed1304df8b99da34-abgc3dca496d510f492c858a2df1eb824e62</Name>
            <Properties>
                <Creation-Time>Fri, 17 Dec 2021 01:04:55 GMT</Creation-Time>
                <Last-Modified>Fri, 17 Dec 2021 01:04:55 GMT</Last-Modified>
                <Etag>0x8D9C0F93D3F6062</Etag>
                <Content-Length>10000</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-CRC64 />
                <Content-MD5>Gvp7eHd5/MlEV1brkD9viQ==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <AccessTier>Hot</AccessTier>
                <AccessTierInferred>true</AccessTierInferred>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>true</ServerEncrypted>
            </Properties>
            <Metadata>
                <Kopiamtime>1577968240000000000</Kopiamtime>
            </Metadata>
            <OrMetadata />
        </Blob>
        <Blob>
            <Name>sastest-1639703094-ed1304df8b99da34-kopia.repository</Name>
            <Properties>
                <Creation-Time>Fri, 17 Dec 2021 01:04:55 GMT</Creation-Time>
                <Last-Modified>Fri, 17 Dec 2021 01:04:55 GMT</Last-Modified>
                <Etag>0x8D9C0F93D5640A2</Etag>
                <Content-Length>100</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-CRC64 />
                <Content-MD5>S3ixhZH4sRD+KIqBPwIHFw==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <AccessTier>Hot</AccessTier>
                <AccessTierInferred>true</AccessTierInferred>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>true</ServerEncrypted>
            </Properties>
            <Metadata>
                <Kopiamtime>1577968240000000000</Kopiamtime>
            </Metadata>
            <OrMetadata />
        </Blob>
        <Blob>
            <Name>sastest-1639703094-ed1304df8b99da34-zxce0e35630770c54668a8cfb4e414c6bf8f</Name>
            <Properties>
                <Creation-Time>Fri, 17 Dec 2021 01:04:55 GMT</Creation-Time>
                <Last-Modified>Fri, 17 Dec 2021 01:04:55 GMT</Last-Modified>
                <Etag>0x8D9C0F93D11C6F2</Etag>
                <Content-Length>1</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-CRC64 />
                <Content-MD5>VaVACK0bpYmqIQ0mKcHfQQ==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <AccessTier>Hot</AccessTier>
                <AccessTierInferred>true</AccessTierInferred>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>true</ServerEncrypted>
            </Properties>
            <Metadata>
                <Kopiamtime>1577968240000000000</Kopiamtime>
            </Metadata>
            <OrMetadata />
        </Blob>
    </Blobs>
    <NextMarker />
</EnumerationResults>

I was looking to retrieve metadata named Kopiamtime which shows up in the server response, but unfortunately cannot be read in Go code, because it.Metadata is always an empty map:

for _, it := range resp.Segment.BlobItems {
  fmt.Println(it.Metadata.AdditionalProperties)
}

This always prints map[].

I'm not an expert in Golang XML handling, but I think this mapping is to blame, probably because of extra nesting:

azure-sdk-for-go/sdk/storage/azblob/zz_generated_models.go

Line 381 in a330cae

Metadata *BlobMetadata `xml:"Metadata"`

Also notice how the server returns OrMetadata but the code is mapping ObjectReplicationMetadata which is also incorrect.

BTW. Thanks for the library. I'm trying to use it in https://github.com/kopia/kopia - all other things are working fine, except this one issue which is currently blocking.

Is there a workaround that can be applied here? I was thinking of manually parsing the XML for the time being until this issue is resolved.

The text was updated successfully, but these errors were encountered:

jkowalski · 2021-12-17T02:05:38Z

In case this helps anybody, here's a quick-and-dirty XML parsing workaround that works and allows me to read the one metadata I'm interested in:

		var enumerationResults struct {
			Blobs struct {
				Blob []struct {
					Name       string
					Properties struct {
						ContentLength int64  `xml:"Content-Length"`
						LastModified  string `xml:"Last-Modified"`
					}
					Metadata struct {
						Kopiamtime string
					}
				}
			}
		}

		if err := xml.NewDecoder(resp.RawResponse.Body).Decode(&enumerationResults); err != nil {
			return errors.Wrap(err, "unable to decode response")
		}

berndverst · 2021-12-18T21:49:18Z

I just came here to report the same issue. For example ContentMD5 property isn't returned from the List Response.

Here is a complete way to reproduce specifically for the ContentMD5 property:

package main

import (
	"context"
	"crypto/md5"
	"encoding/base64"
	"fmt"
	"os"

	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
)

func main() {
	cred, err := azblob.NewSharedKeyCredential(os.Getenv("AzureBlobStorageAccount"), os.Getenv("AzureBlobStorageAccessKey"))
	if err != nil {
		fmt.Printf("failed to create credential %v", err)
	}
	url := fmt.Sprintf("https://%s.blob.core.windows.net/%s", os.Getenv("AzureBlobStorageAccount"), os.Getenv("AzureBlobStorageContainer"))
	container, err := azblob.NewContainerClientWithSharedKey(url, cred, nil)
	if err != nil {
		fmt.Printf("failed to create container client %v", err)
	}
	blockBlob := container.NewBlockBlobClient("randomfile.txt")
	content := []byte("Hello World!")
	md5hash := md5.New()
	md5hash.Write(content)
	b64md5hash := base64.StdEncoding.EncodeToString(md5hash.Sum(nil))

	blockBlob.UploadBufferToBlockBlob(context.Background(), content, azblob.HighLevelUploadToBlockBlobOption{
		HTTPHeaders: &azblob.BlobHTTPHeaders{
			BlobContentMD5: []byte(b64md5hash),
		},
	})
	props, _ := blockBlob.GetProperties(context.Background(), nil)

	receivedB64md5hash := base64.StdEncoding.EncodeToString(props.ContentMD5)
	if b64md5hash != receivedB64md5hash {
		panic("MD5 hash mismatch")
	}
	fmt.Println("MD5 hash from GetProperties response verified")

	pager := container.ListBlobsFlat(nil)

	blobs := []*azblob.BlobItemInternal{}
	for pager.NextPage(context.Background()) {
		response := pager.PageResponse()
		blobs = append(blobs, response.ContainerListBlobFlatSegmentResult.Segment.BlobItems...)
	}

	for _, blob := range blobs {
		if *blob.Name == "randomfile.txt" {
			receivedListb64md5Hash := base64.StdEncoding.EncodeToString(blob.Properties.ContentMD5)

			// blob.Properties.ContentMD5 is nil
			// this seems to be an issue marshalling the response from the server, but in the List request only
			if b64md5hash != receivedListb64md5Hash {
				fmt.Println("MD5 hash from ListBlobsFlat response does not match")
				panic("MD5 hash mismatch")
			}
			break
		}
	}
}

berndverst · 2021-12-18T21:52:05Z

@RickWinter @jhendrixMSFT FYI I am currently rewriting the Dapr Blob Storage Output Binding component to use the Track 2 SDK and I encountered this issue in my tests.

Until the List response contains all the properties I will not be able to certify this component as Stable as far as Dapr is concerned.

berndverst · 2021-12-18T22:05:30Z

Thanks for the tip @jkowalski. I am not sure why the original code doesn't Decode the XML correctly, but the following seems to work for me with regards to the ContentMD5 property

The following code depends on my code snippet from above.

var enumerationResults struct {
		Blobs struct {
			Blob []struct {
				Name       string `xml:"Name"`
				Properties struct {
					ContentMD5 []byte `xml:"Content-MD5"`
				}
			}
		}
	}

	for pager.NextPage(context.Background()) {
		response := pager.PageResponse()

		if err := xml.NewDecoder(response.RawResponse.Body).Decode(&enumerationResults); err != nil {
			panic(err)
		}

		for _, blob := range enumerationResults.Blobs.Blob {
			if blob.Name == "randomfile.txt" {
				receivedListb64md5Hash := blob.Properties.ContentMD5
				if b64md5hash != string(receivedListb64md5Hash) {
					fmt.Println("MD5 hash from ListBlobsFlat response does not match")
					panic("MD5 hash mismatch")
				}
				fmt.Println("MD5 hash from ListBlobsFlat response verified")
				break
			}
		}

	}

deiter · 2022-01-14T22:50:18Z

Hello,

I have the same issue. Could you please review my fix:

--- azure-sdk-for-go/sdk/storage/azblob/zz_generated_models.go 2022-01-14 22:44:54.382333817 +0000
+++ azure-sdk-for-go/sdk/storage/azblob/zz_generated_models.go 2022-01-14 22:44:29.330518663 +0000
@@ -512,6 +512,9 @@
        b.DeletedTime = (*time.Time)(aux.DeletedTime)
        b.ExpiresOn = (*time.Time)(aux.ExpiresOn)
        b.LastModified = (*time.Time)(aux.LastModified)
+       if aux.ContentMD5 != nil {
+               b.ContentMD5 = *aux.ContentMD5
+       }
        return nil
 }

and the blob properties is:

Properties &{Etag:0xc000365250 LastModified:2022-01-14 22:23:04 +0000 GMT AccessTier:0xc0003652c0 AccessTierChangeTime:<nil> AccessTierInferred:0xc00021bacd ArchiveStatus:<nil> BlobSequenceNumber:<nil> BlobType:0xc0003652b0 CacheControl:0xc000365290 ContentDisposition:0xc0003652a0 ContentEncoding:0xc000365270 ContentLanguage:0xc000365280 ContentLength:0xc00021b9a8 ContentMD5:[106 88 100 47 79 70 48 57 47 115 105 66 88 83 68 51 83 87 65 109 51 65 61 61] ContentType:0xc000365260 CopyCompletionTime:<nil> CopyID:<nil> CopyProgress:<nil> CopySource:<nil> CopyStatus:<nil> CopyStatusDescription:<nil> CreationTime:2022-01-14 21:24:17 +0000 GMT CustomerProvidedKeySHA256:<nil> DeletedTime:<nil> DestinationSnapshot:<nil> EncryptionScope:<nil> ExpiresOn:<nil> IncrementalCopy:<nil> IsSealed:<nil> LeaseDuration:<nil> LeaseState:0xc0003652e0 LeaseStatus:0xc0003652d0 RehydratePriority:<nil> RemainingRetentionDays:<nil> ServerEncrypted:0xc00021bb59 TagCount:<nil>}

Thank you!

deiter · 2022-01-15T00:52:51Z

And of course we can decode base64 to []byte:

        if aux.ContentMD5 != nil {
                contentMD5, err := base64.StdEncoding.DecodeString(*aux.ContentMD5)
                if err != nil {
                        return err
                }
                b.ContentMD5 = contentMD5
        }

evenh · 2022-01-27T20:40:13Z

I'm encountering this issue as well. Had to apply @deiter's fix to a fork of mine in order to get Content-MD5 in properties response. How can we in the community best help in order to get this bug fixed?

Workaround for Azure#16679

jhendrixMSFT · 2022-02-09T15:55:29Z

Fixed in sdk/storage/azblob/v0.3.0

evenh · 2022-02-10T11:20:27Z

0.3.0 doesn't seem different in this regard than 0.2.0 to me

mohsha-msft · 2022-02-10T11:25:59Z

Are you able to reproduce this issue again @evenh ?

Here's a test which I added to make sure this issue isn't happening again.

azure-sdk-for-go/sdk/storage/azblob/zt_container_client_test.go

Line 1234 in a616d27

func (s *azblobTestSuite) TestListBlobIncludeMetadata() {

evenh · 2022-02-10T11:29:24Z

I see that I wasn't specific enough about mye use case.

We use AzCopy to upload a bunch of files (which puts MD5 in the metadata) and we'd like to access that property via this SDK. My testing suggests that the Content-MD5 property always is []. I've fixed it temporarily using @deiter's fix in evenh@5d716a1 for my own purposes.

I couldn't figure out to run the tests locally, so haven't tried them.

jhendrixMSFT · 2022-02-10T15:42:52Z

This is a codegen issue, I've opened Azure/autorest.go#774 to track it.

For now, @mohsha-msft can you fix this up manually? I don't know how many locations we improperly unmarshal base-64 strings.

evenh · 2022-02-10T15:53:46Z

Thanks 👏🏻

mohsha-msft · 2022-02-10T16:59:15Z

Thanks Joel. I'll keep track of this. I'll fix it manually for now. Do we have any transform to fix this?

jhendrixMSFT · 2022-02-10T17:27:22Z

I don't think it can be fixed via a transform.

mohsha-msft · 2022-04-20T05:52:33Z

Hey @jkowalski , @berndverst , @evenh , and @deiter,

azblob v0.4.0 is now publically available. I have fixed the issue here and here.

Please reach out and reopen the issue if it still persists.

Thanks a lot for your feedbacks!

jhendrixMSFT assigned mohsha-msft and adreed-msft Dec 17, 2021

jhendrixMSFT added the Storage Storage Service (Queues, Blobs, Files) label Dec 17, 2021

ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Dec 17, 2021

RickWinter added the Client This issue points to a problem in the data-plane of the library. label Dec 17, 2021

ghost added the needs-team-attention This issue needs attention from Azure service team or SDK team label Dec 17, 2021

RickWinter added bug This issue requires a change to an existing behavior in the product in order to be resolved. and removed question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Dec 17, 2021

RickWinter added this to the [2022] January milestone Dec 17, 2021

RickWinter assigned zezha-msft Jan 14, 2022

RickWinter modified the milestones: [2022] January, [2022] February Jan 14, 2022

evenh added a commit to evenh/azure-sdk-for-go that referenced this issue Jan 28, 2022

Return Content-MD5 for blob properties

5d716a1

Workaround for Azure#16679

mohsha-msft closed this as completed Feb 9, 2022

mohsha-msft reopened this Feb 10, 2022

RickWinter modified the milestones: [2022] February, [2022] March Feb 24, 2022

This was referenced Feb 24, 2022

[Azure][Track 2 SDK] Migrate Azure Blob Storage Binding dapr/components-contrib#1530

Closed

[Azure][Track 2 SDK] Migrate Azure Blobstorage State Store dapr/components-contrib#1529

Closed

mohsha-msft closed this as completed Apr 20, 2022

RickWinter removed the needs-team-attention This issue needs attention from Azure service team or SDK team label Apr 20, 2022

github-actions bot locked and limited conversation to collaborators Apr 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure Blob: Unable to access Metadata when listing blobs #16679

Azure Blob: Unable to access Metadata when listing blobs #16679

jkowalski commented Dec 17, 2021

jkowalski commented Dec 17, 2021 •

edited

berndverst commented Dec 18, 2021

berndverst commented Dec 18, 2021

berndverst commented Dec 18, 2021 •

edited

deiter commented Jan 14, 2022

deiter commented Jan 15, 2022

evenh commented Jan 27, 2022 •

edited

jhendrixMSFT commented Feb 9, 2022

evenh commented Feb 10, 2022

mohsha-msft commented Feb 10, 2022

evenh commented Feb 10, 2022 •

edited

jhendrixMSFT commented Feb 10, 2022

evenh commented Feb 10, 2022

mohsha-msft commented Feb 10, 2022

jhendrixMSFT commented Feb 10, 2022

mohsha-msft commented Apr 20, 2022

Azure Blob: Unable to access Metadata when listing blobs #16679

Azure Blob: Unable to access Metadata when listing blobs #16679

Comments

jkowalski commented Dec 17, 2021

Bug Report

jkowalski commented Dec 17, 2021 • edited

berndverst commented Dec 18, 2021

berndverst commented Dec 18, 2021

berndverst commented Dec 18, 2021 • edited

deiter commented Jan 14, 2022

deiter commented Jan 15, 2022

evenh commented Jan 27, 2022 • edited

jhendrixMSFT commented Feb 9, 2022

evenh commented Feb 10, 2022

mohsha-msft commented Feb 10, 2022

evenh commented Feb 10, 2022 • edited

jhendrixMSFT commented Feb 10, 2022

evenh commented Feb 10, 2022

mohsha-msft commented Feb 10, 2022

jhendrixMSFT commented Feb 10, 2022

mohsha-msft commented Apr 20, 2022

jkowalski commented Dec 17, 2021 •

edited

berndverst commented Dec 18, 2021 •

edited

evenh commented Jan 27, 2022 •

edited

evenh commented Feb 10, 2022 •

edited