GetBucketLifecycleConfiguration deserialization failed #1941
Comments
microyahoo
added
bug
|
|
|
Hi @microyahoo , I'm not sure why you are running into this error. I'm able to successfully put and retrieve a lifecycle configuration on my bucket. Example: func main() {
bucketname := "foo-bucket"
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("us-east-1"), config.WithClientLogMode(aws.LogResponseWithBody|aws.LogRequestWithBody))
if err != nil {
log.Fatalf("unable to load SDK config, %v", err)
}
client := s3.NewFromConfig(cfg)
_, err = client.PutBucketLifecycleConfiguration(context.Background(), &s3.PutBucketLifecycleConfigurationInput{
Bucket: aws.String(bucketname),
LifecycleConfiguration: &types.BucketLifecycleConfiguration{
Rules: []types.LifecycleRule{
{
ID: aws.String("myrule-1"),
Filter: &types.LifecycleRuleFilterMemberObjectSizeGreaterThan{
Value: 1024,
},
NoncurrentVersionExpiration: &types.NoncurrentVersionExpiration{NoncurrentDays: 3},
Expiration: &types.LifecycleExpiration{
Days: 1,
},
Status: "Enabled",
},
},
},
})
if err != nil {
panic(err)
}
out, err := client.GetBucketLifecycleConfiguration(context.Background(),
&s3.GetBucketLifecycleConfigurationInput{
Bucket: aws.String(bucketname),
})
if err != nil {
panic(err)
}
fmt.Println("found: ", len(out.Rules), "rule(s).")
}output: $ go run main.go
<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration
xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>myrule-1</ID>
<Filter>
<ObjectSizeGreaterThan>1024</ObjectSizeGreaterThan>
</Filter>
<Status>Enabled</Status>
<Expiration>
<Days>1</Days>
</Expiration>
<NoncurrentVersionExpiration>
<NoncurrentDays>3</NoncurrentDays>
</NoncurrentVersionExpiration>
</Rule>
</LifecycleConfiguration>
found: 1 rule(s).My guess is that somehow you uploaded a malformed XML describing the lifecycle configuration. That is you are running into this deserialization error. Hope my code example helps. |
RanVaknin
added
response-requested
|
Hi @RanVaknin, thanks for your response. The reason for input := &s3.PutBucketLifecycleConfigurationInput{
Bucket: aws.String(bucket),
LifecycleConfiguration: &types.BucketLifecycleConfiguration{
Rules: []types.LifecycleRule{
types.LifecycleRule{
Status: types.ExpirationStatusEnabled,
ID: aws.String("rule-uuid1"),
// Filter: &types.LifecycleRuleFilterMemberPrefix{},
Expiration: &types.LifecycleExpiration{
Days: 2,
},
},
},
},
} |
|
In addition, the following example with replicated package main
import (
"context"
"fmt"
"log"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
"github.com/aws/aws-sdk-go/aws/awsutil"
)
func main() {
const (
accessKey = "REDACTED"
secretKey = "REDACTED"
bucket = "REDACTED"
endpoint = "http://127.0.0.1:80"
)
appCreds := aws.NewCredentialsCache(credentials.NewStaticCredentialsProvider(
accessKey, secretKey, ""))
cfg, err := config.LoadDefaultConfig(context.TODO(),
config.WithCredentialsProvider(appCreds),
config.WithEndpointResolver(
aws.EndpointResolverFunc(
func(service, region string) (aws.Endpoint, error) {
return aws.Endpoint{
Source: aws.EndpointSourceCustom,
URL: endpoint,
}, nil
})),
)
if err != nil {
log.Fatal(err)
}
// Create an S3 service client
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.UsePathStyle = true
})
// lifecycle
ctx := context.Background()
lcc, err := client.GetBucketLifecycleConfiguration(ctx,
&s3.GetBucketLifecycleConfigurationInput{
Bucket: aws.String(bucket),
})
if err != nil {
log.Println(err)
}
fmt.Printf("get bucket lifecycle configuration: %s\n", awsutil.Prettify(lcc))
input := &s3.PutBucketLifecycleConfigurationInput{
Bucket: aws.String(bucket),
LifecycleConfiguration: &types.BucketLifecycleConfiguration{
Rules: []types.LifecycleRule{
types.LifecycleRule{
Status: types.ExpirationStatusEnabled,
ID: aws.String("rule-uuid1"),
Filter: &types.LifecycleRuleFilterMemberAnd{
Value: types.LifecycleRuleAndOperator{
Prefix: aws.String("prefix"),
Tags: []types.Tag{
{
Key: aws.String("color"),
Value: aws.String("red"),
},
{
Key: aws.String("color"),
Value: aws.String("blue"),
},
},
},
},
Expiration: &types.LifecycleExpiration{
Days: 2,
},
},
},
},
}
fmt.Printf("lifecycle input: %s\n", awsutil.Prettify(input))
lccOutput, err := client.PutBucketLifecycleConfiguration(ctx, input)
if err != nil {
log.Fatal(err)
}
fmt.Printf("lifecycle configuration output: %s\n", awsutil.Prettify(lccOutput))
lcc, err = client.GetBucketLifecycleConfiguration(ctx,
&s3.GetBucketLifecycleConfigurationInput{
Bucket: aws.String(bucket),
})
if err != nil {
log.Fatal(err)
}
fmt.Printf("get lifecycle configuration: %s\n", awsutil.Prettify(lcc))
}output: 🍺 /root/go/src/github.com/microyahoo/go-exercises/s3-examples ☞ git:(master) ✗ go run test_s3.go
2022/11/29 14:20:07 operation error S3: GetBucketLifecycleConfiguration, https response error StatusCode: 404, RequestID: tx00000cc56de0884bdea0b-006385a497-133cd7-os-ztjyqxatbzfjbn8t, HostID: 133cd7-os-ztjyqxatbzfjbn8t-os-ztjyqxatbzfjbn8t, api error NoSuchLifecycleConfiguration: UnknownError
get bucket lifecycle configuration: <invalid value>
lifecycle input: {
Bucket: "zhengliang",
ChecksumAlgorithm: ,
LifecycleConfiguration: {
Rules: [{
Status: Enabled,
Expiration: {
Days: 2,
ExpiredObjectDeleteMarker: false
},
Filter: &{{0 0 0xc00011a280 [{0xc00011a290 0xc00011a2a0 {}} {0xc00011a2b0 0xc00011a2c0 {}}] {}} {}},
ID: "rule-uuid1"
}]
}
}
lifecycle configuration output: {
ResultMetadata: {
}
}
2022/11/29 14:20:07 operation error S3: GetBucketLifecycleConfiguration, https response error StatusCode: 200, RequestID: tx00000dcbe144593552126-006385a497-133cd7-os-ztjyqxatbzfjbn8t, HostID: , deserialization failed, failed to decode response body, EOF
exit status 1After running the example, view the lifecycle through ➜ /root/go/src/github.com/microyahoo/go-exercises/s3-examples ☞ git:(master) ✗ s3cmd getlifecycle s3://zhengliang
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>rule-uuid1</ID>
<Filter>
<Prefix>prefix</Prefix>
<Tag>
<Key>color</Key>
<Value>red</Value>
</Tag>
</Filter>
<Status>Enabled</Status>
<Expiration>
<Days>2</Days>
</Expiration>
</Rule>
</LifecycleConfiguration> |
|
You included your secret and access key. I have redacted them from your code snippet. For your own security please omit that info in the future. Regarding your code. Again, I'm not sure how you are able to I'm really confused as to how this executes more than anything. I'd say this is why you are running into a malformed XML. When I omit the duplicate I get a valid response: <?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration
xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>rule-uuid1</ID>
<Filter>
<And>
<Prefix>prefix</Prefix>
<Tag>
<Key>color</Key>
<Value>red</Value>
</Tag>
</And>
</Filter>
<Status>Enabled</Status>
<Expiration>
<Days>2</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>From the CLI it doesn't seem like you are getting the same policy as the one shown in your code. Your Also, I would enable request and response logging on your client. That will give you some more information on what is being sent and received over the wire so you can further debug this behavior. You can do that by including these lines in your config: cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("us-east-1"), config.WithClientLogMode(aws.LogResponseWithBody|aws.LogRequestWithBody))Please let me know if that helps. Thank you, |
RanVaknin
added
response-requested
Hi @RanVaknin, thanks for you reminder. I will pay attention later.
The underlying object storage I use is
No, I create it using sdk.
The following is output after including config |
Please help to confirm why does |
|
Hi @microyahoo , I deleted your logs because it had your credentials in them and were not redacted 🥲 . Before I deleted them I saw that after you fixed the duplicate tag you got a 200. So I assume that this part is solved. Regarding your question:
From the S3 docs:
Regarding you using Let me know if you need anything else. |
github-actions
bot
removed
the
response-requested
|
I have responded on this discussion thread with my findings. Please refer to it for clarifications. |
RanVaknin
added
the
response-requested
|
Thanks for your nice explanation, I understand now. About ceph radosgw, please refer the link radosgw |
|
Describe the bug
2022/11/25 13:34:13 operation error S3: GetBucketLifecycleConfiguration, https response error StatusCode: 200, RequestID: tx00000955d58f77184c2cb-00638053d5-3813-os-ztjyqxatbzfjbn8t, HostID: , deserialization failed, failed to decode response body, EOFs3cmdis OK to get lifecycleExpected Behavior
return no error
Current Behavior
deserialization failed, failed to decode response body, EOF
Reproduction Steps
Possible Solution
No response
Additional Information/Context
No response
AWS Go SDK V2 Module Versions Used
Compiler and Version used
go version go1.19.3 linux/amd64
Operating System and version
Linux k1 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
The text was updated successfully, but these errors were encountered: