Specifying a custom port in host_base cases V4 signature error

[jamshid]

I test s3cmd against an S3-compatible service and sometimes the test system's S3 endpoint uses a port that is not the default 80 or 443.

Since I configure ~/.s3cfg with a script during testing I usually included the port number in host_base / host_bucket, even if it is the default, e.g.

access_key = X
secret_key = Y
host_base = s3.amazonaws.com:443
host_bucket = %(bucket)s.s3.amazonaws.com:443
#signature_v2=True

I just discovered that only works as long as you're using signature_v2=True. If you comment that out and use V4 signatures then s3cmd fails with a 403 SignatureDoesNotMatch.

Is the problem related to the Host header? Is there another way that the port number should be specified?

DEBUG: s3cmd version 2.0.1
DEBUG: ConfigParser: Reading file '/Users/jamshid/.s3cfg'
DEBUG: ConfigParser: access_key->AK...17_chars...Q
DEBUG: ConfigParser: secret_key->gB...37_chars...8
DEBUG: ConfigParser: host_base->s3.amazonaws.com:443
DEBUG: ConfigParser: host_bucket->%(bucket)s.s3.amazonaws.com:443
DEBUG: Updating Config.Config cache_file -> 
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'ls' using UTF-8
DEBUG: Unicodising 's3://mahbucke' using UTF-8
DEBUG: Command: ls
DEBUG: Bucket 's3://mahbucke':
DEBUG: CreateRequest: resource[uri]=/
DEBUG: ===== Send_request inner request to determine the bucket region =====
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(mahbucke): s3.amazonaws.com:443
DEBUG: canonical_headers = host:s3.amazonaws.com:443
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180521T195540Z

DEBUG: Canonical Request:
GET
/mahbucke/
location=
host:s3.amazonaws.com:443
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180521T195540Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': u'AWS4-HMAC-SHA256 Credential=AKIAJV3VAEW6OFKXWMLQ/20180521/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=12e3173e1f135bffb9aa6532f7b02ab837f14d6ea618bcd70f97503921a3fccb', 'x-amz-date': '20180521T195540Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(mahbucke): s3.amazonaws.com:443
DEBUG: ConnMan.get(): creating new connection: https://s3.amazonaws.com:443
DEBUG: Using ca_certs_file None
DEBUG: httplib.HTTPSConnection() has only context
DEBUG: non-proxied HTTPSConnection(s3.amazonaws.com, 443)
DEBUG: format_uri(): /mahbucke/?location
DEBUG: Sending request method_string='GET', uri=u'/mahbucke/?location', headers={'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': u'AWS4-HMAC-SHA256 Credential=AKIAJV3VAEW6OFKXWMLQ/20180521/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=12e3173e1f135bffb9aa6532f7b02ab837f14d6ea618bcd70f97503921a3fccb', 'x-amz-date': '20180521T195540Z'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://s3.amazonaws.com:443#1)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAJV3VAEW6OFKXWMLQ</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256\n20180521T195540Z\n20180521/us-east-1/s3/aws4_request\nbf46c17714b7176b4a3c3371e47065b064b9769b6aa0d3caabd3aec136a5ddbe</StringToSign><SignatureProvided>12e3173e1f135bffb9aa6532f7b02ab837f14d6ea618bcd70f97503921a3fccb</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 38 30 35 32 31 54 31 39 35 35 34 30 5a 0a 32 30 31 38 30 35 32 31 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 62 66 34 36 63 31 37 37 31 34 62 37 31 37 36 62 34 61 33 63 33 33 37 31 65 34 37 30 36 35 62 30 36 34 62 39 37 36 39 62 36 61 61 30 64 33 63 61 61 62 64 33 61 65 63 31 33 36 61 35 64 64 62 65</StringToSignBytes><CanonicalRequest>GET\n/mahbucke/\nlocation=\nhost:s3.amazonaws.com\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20180521T195540Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 6d 61 68 62 75 63 6b 65 2f 0a 6c 6f 63 61 74 69 6f 6e 3d 0a 68 6f 73 74 3a 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 38 30 35 32 31 54 31 39 35 35 34 30 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes><RequestId>A2FDAFD3B3075A36</RequestId><HostId>Looy+daMa7E0Q+ICuAEUM9mtkh2YKhuvTZLm2C5DeIc7pl+DH8QbwwcK/dF3Vg/wSHhca3n7G+4=</HostId></Error>',
 'headers': {'content-type': 'application/xml',
             'date': 'Mon, 21 May 2018 19:55:39 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-id-2': 'Looy+daMa7E0Q+ICuAEUM9mtkh2YKhuvTZLm2C5DeIc7pl+DH8QbwwcK/dF3Vg/wSHhca3n7G+4=',
             'x-amz-request-id': 'A2FDAFD3B3075A36'},
 'reason': 'Forbidden',
 'status': 403}
DEBUG: S3Error: 403 (Forbidden)
DEBUG: HttpHeader: x-amz-id-2: Looy+daMa7E0Q+ICuAEUM9mtkh2YKhuvTZLm2C5DeIc7pl+DH8QbwwcK/dF3Vg/wSHhca3n7G+4=
DEBUG: HttpHeader: server: AmazonS3
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: x-amz-request-id: A2FDAFD3B3075A36
DEBUG: HttpHeader: date: Mon, 21 May 2018 19:55:39 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch'
DEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.'
DEBUG: ErrorXML: AWSAccessKeyId: 'AKIAJV3VAEW6OFKXWMLQ'
DEBUG: ErrorXML: StringToSign: 'AWS4-HMAC-SHA256\n20180521T195540Z\n20180521/us-east-1/s3/aws4_request\nbf46c17714b7176b4a3c3371e47065b064b9769b6aa0d3caabd3aec136a5ddbe'
DEBUG: ErrorXML: SignatureProvided: '12e3173e1f135bffb9aa6532f7b02ab837f14d6ea618bcd70f97503921a3fccb'
DEBUG: ErrorXML: StringToSignBytes: '41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 38 30 35 32 31 54 31 39 35 35 34 30 5a 0a 32 30 31 38 30 35 32 31 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 62 66 34 36 63 31 37 37 31 34 62 37 31 37 36 62 34 61 33 63 33 33 37 31 65 34 37 30 36 35 62 30 36 34 62 39 37 36 39 62 36 61 61 30 64 33 63 61 61 62 64 33 61 65 63 31 33 36 61 35 64 64 62 65'
DEBUG: ErrorXML: CanonicalRequest: 'GET\n/mahbucke/\nlocation=\nhost:s3.amazonaws.com\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20180521T195540Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
DEBUG: ErrorXML: CanonicalRequestBytes: '47 45 54 0a 2f 6d 61 68 62 75 63 6b 65 2f 0a 6c 6f 63 61 74 69 6f 6e 3d 0a 68 6f 73 74 3a 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 38 30 35 32 31 54 31 39 35 35 34 30 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35'
DEBUG: ErrorXML: RequestId: 'A2FDAFD3B3075A36'
DEBUG: ErrorXML: HostId: 'Looy+daMa7E0Q+ICuAEUM9mtkh2YKhuvTZLm2C5DeIc7pl+DH8QbwwcK/dF3Vg/wSHhca3n7G+4='
DEBUG: DeUnicodising u'403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method.' using UTF-8
DEBUG: Error getlocation inner request: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method.
DEBUG: Using signature v4
DEBUG: get_hostname(mahbucke): mahbucke.s3.amazonaws.com:443
DEBUG: canonical_headers = host:mahbucke.s3.amazonaws.com:443
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180521T195540Z

DEBUG: Canonical Request:
GET
/
delimiter=%2F
host:mahbucke.s3.amazonaws.com:443
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180521T195540Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': u'AWS4-HMAC-SHA256 Credential=AKIAJV3VAEW6OFKXWMLQ/20180521/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=340643d77851eb0becb538462741818e06e316019e795d9313fc112912c56b86', 'x-amz-date': '20180521T195540Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(mahbucke): mahbucke.s3.amazonaws.com:443
DEBUG: ConnMan.get(): creating new connection: https://mahbucke.s3.amazonaws.com:443
DEBUG: httplib.HTTPSConnection() has only context
DEBUG: non-proxied HTTPSConnection(mahbucke.s3.amazonaws.com, 443)
DEBUG: format_uri(): /?delimiter=%2F
DEBUG: Sending request method_string='GET', uri=u'/?delimiter=%2F', headers={'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': u'AWS4-HMAC-SHA256 Credential=AKIAJV3VAEW6OFKXWMLQ/20180521/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=340643d77851eb0becb538462741818e06e316019e795d9313fc112912c56b86', 'x-amz-date': '20180521T195540Z'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://mahbucke.s3.amazonaws.com:443#1)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region \'US\' is wrong; expecting \'us-east-1\'</Message><Region>us-east-1</Region><RequestId>1B2C9729C3BCFAC9</RequestId><HostId>+j8BJF+Yd6ixsOzNwGmxctlYjnp8VVxTVqwADciytRcbsNsiH7Z4xIHTsHG5/UV0njBQvNnZhSU=</HostId></Error>',
 'headers': {'connection': 'close',
             'content-type': 'application/xml',
             'date': 'Mon, 21 May 2018 19:55:40 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-bucket-region': 'us-east-1',
             'x-amz-id-2': '+j8BJF+Yd6ixsOzNwGmxctlYjnp8VVxTVqwADciytRcbsNsiH7Z4xIHTsHG5/UV0njBQvNnZhSU=',
             'x-amz-request-id': '1B2C9729C3BCFAC9'},
 'reason': 'Bad Request',
 'status': 400}
INFO: Forwarding request to us-east-1
DEBUG: Using signature v4
DEBUG: get_hostname(mahbucke): mahbucke.s3.amazonaws.com:443
DEBUG: canonical_headers = host:mahbucke.s3.amazonaws.com:443
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180521T195540Z

DEBUG: Canonical Request:
GET
/
delimiter=%2F
host:mahbucke.s3.amazonaws.com:443
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180521T195540Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'x-amz-date': '20180521T195540Z', 'Authorization': u'AWS4-HMAC-SHA256 Credential=AKIAJV3VAEW6OFKXWMLQ/20180521/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=579e17be2471bb7ec8295221e38d46ae0458b7682d18a5287e427f6fbe87e6fc'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(mahbucke): mahbucke.s3.amazonaws.com:443
DEBUG: ConnMan.get(): re-using connection: https://mahbucke.s3.amazonaws.com:443#1
DEBUG: format_uri(): /?delimiter=%2F
DEBUG: Sending request method_string='GET', uri=u'/?delimiter=%2F', headers={'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'x-amz-date': '20180521T195540Z', 'Authorization': u'AWS4-HMAC-SHA256 Credential=AKIAJV3VAEW6OFKXWMLQ/20180521/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=579e17be2471bb7ec8295221e38d46ae0458b7682d18a5287e427f6fbe87e6fc'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://mahbucke.s3.amazonaws.com:443#2)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAJV3VAEW6OFKXWMLQ</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256\n20180521T195540Z\n20180521/us-east-1/s3/aws4_request\n20a128cee803c681d5ac1d6384e0d0c5b2fb587deba29bdc8a8bb680770a283d</StringToSign><SignatureProvided>579e17be2471bb7ec8295221e38d46ae0458b7682d18a5287e427f6fbe87e6fc</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 38 30 35 32 31 54 31 39 35 35 34 30 5a 0a 32 30 31 38 30 35 32 31 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 32 30 61 31 32 38 63 65 65 38 30 33 63 36 38 31 64 35 61 63 31 64 36 33 38 34 65 30 64 30 63 35 62 32 66 62 35 38 37 64 65 62 61 32 39 62 64 63 38 61 38 62 62 36 38 30 37 37 30 61 32 38 33 64</StringToSignBytes><CanonicalRequest>GET\n/\ndelimiter=%2F\nhost:mahbucke.s3.amazonaws.com\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20180521T195540Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 0a 64 65 6c 69 6d 69 74 65 72 3d 25 32 46 0a 68 6f 73 74 3a 6d 61 68 62 75 63 6b 65 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 38 30 35 32 31 54 31 39 35 35 34 30 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes><RequestId>A976C2F948B4EBC3</RequestId><HostId>WrwxImmb8HWPIu9hMs37d8JOePJKjtoSoAOdR0aX/q2ecTVebOGVD6eQGK0TGyzEEB3UBujm30k=</HostId></Error>',
 'headers': {'content-type': 'application/xml',
             'date': 'Mon, 21 May 2018 19:55:40 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-bucket-region': 'us-east-1',
             'x-amz-id-2': 'WrwxImmb8HWPIu9hMs37d8JOePJKjtoSoAOdR0aX/q2ecTVebOGVD6eQGK0TGyzEEB3UBujm30k=',
             'x-amz-request-id': 'A976C2F948B4EBC3'},
 'reason': 'Forbidden',
 'status': 403}
DEBUG: S3Error: 403 (Forbidden)
DEBUG: HttpHeader: x-amz-bucket-region: us-east-1
DEBUG: HttpHeader: x-amz-id-2: WrwxImmb8HWPIu9hMs37d8JOePJKjtoSoAOdR0aX/q2ecTVebOGVD6eQGK0TGyzEEB3UBujm30k=
DEBUG: HttpHeader: server: AmazonS3
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: x-amz-request-id: A976C2F948B4EBC3
DEBUG: HttpHeader: date: Mon, 21 May 2018 19:55:40 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch'
DEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.'
DEBUG: ErrorXML: AWSAccessKeyId: 'AKIAJV3VAEW6OFKXWMLQ'
DEBUG: ErrorXML: StringToSign: 'AWS4-HMAC-SHA256\n20180521T195540Z\n20180521/us-east-1/s3/aws4_request\n20a128cee803c681d5ac1d6384e0d0c5b2fb587deba29bdc8a8bb680770a283d'
DEBUG: ErrorXML: SignatureProvided: '579e17be2471bb7ec8295221e38d46ae0458b7682d18a5287e427f6fbe87e6fc'
DEBUG: ErrorXML: StringToSignBytes: '41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 38 30 35 32 31 54 31 39 35 35 34 30 5a 0a 32 30 31 38 30 35 32 31 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 32 30 61 31 32 38 63 65 65 38 30 33 63 36 38 31 64 35 61 63 31 64 36 33 38 34 65 30 64 30 63 35 62 32 66 62 35 38 37 64 65 62 61 32 39 62 64 63 38 61 38 62 62 36 38 30 37 37 30 61 32 38 33 64'
DEBUG: ErrorXML: CanonicalRequest: 'GET\n/\ndelimiter=%2F\nhost:mahbucke.s3.amazonaws.com\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20180521T195540Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
DEBUG: ErrorXML: CanonicalRequestBytes: '47 45 54 0a 2f 0a 64 65 6c 69 6d 69 74 65 72 3d 25 32 46 0a 68 6f 73 74 3a 6d 61 68 62 75 63 6b 65 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 38 30 35 32 31 54 31 39 35 35 34 30 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35'
DEBUG: ErrorXML: RequestId: 'A976C2F948B4EBC3'
DEBUG: ErrorXML: HostId: 'WrwxImmb8HWPIu9hMs37d8JOePJKjtoSoAOdR0aX/q2ecTVebOGVD6eQGK0TGyzEEB3UBujm30k='
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method.

[fviard]

Aws signature documentation states that the "port" should not be included for the host field used for signature if it is standard ports 80 for http and 443 for https.

Sadly, there no way to test with amazon that if we were to use another port, the port info would be needed.

I might add in the signature a directive like: if port = 80 or 443 then "strip it".
But i'm not sure that it would be a good idea...

[samuelsh]

Actually, this is what Boto team did.
They stripping standard port from host header:
boto/botocore#1296

As well AWS cli:
aws/aws-cli#2883

[fviard]

Hum, I need to check but I think that I already fixed the issue already. Le mar. 31 juil. 2018 18:13, samuelsh <notifications@github.com> a écrit :

…

Actually, this is what boto team did. The stripping standard port from host header: boto/botocore#1296 <boto/botocore#1296> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#974 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABAUpFVMdUizQqEOz05mXr_2WylPJm70ks5uMIIogaJpZM4UHiKs> .

[jamshid]

No, I just verified the problem still exists in 2.0.2 and in latest source. Removing :80 from ~/.s3cfg works around the problem.

# s3cmd -d  ls s3://mybucket
DEBUG: s3cmd version 2.0.2
DEBUG: ConfigParser: Reading file '/root/.s3cfg'
DEBUG: ConfigParser: access_key->d6...29_chars...1
DEBUG: ConfigParser: secret_key->se...3_chars...t
DEBUG: ConfigParser: host_bucket->%(bucket)s.backup67:80
DEBUG: ConfigParser: host_base->backup67:80
DEBUG: ConfigParser: use_https->false
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: check_ssl_certificate->False
DEBUG: ConfigParser: socket_timeout->600
DEBUG: Updating Config.Config cache_file -> 
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'ls' using ANSI_X3.4-1968
DEBUG: Unicodising 's3://mybucket' using ANSI_X3.4-1968
DEBUG: Command: ls
DEBUG: Bucket 's3://mybucket':
DEBUG: CreateRequest: resource[uri]=/
DEBUG: ===== Send_request inner request to determine the bucket region =====
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(mybucket): backup67:80
DEBUG: canonical_headers = host:backup67:80
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20191014T175314Z

DEBUG: Canonical Request:
GET
/mybucket/
location=
host:backup67:80
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20191014T175314Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': u'AWS4-HMAC-SHA256 Credential=d6d2206a5638cca2b3d181fe156f8921/20191014/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=6ff26f7f6ac5bc0a8aa245e464f3929d019ceb65edb80d7cf8ddc6084fe9bdd7', 'x-amz-date': '20191014T175314Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(mybucket): backup67:80
DEBUG: ConnMan.get(): creating new connection: http://backup67:80
DEBUG: non-proxied HTTPConnection(backup67, 80)
DEBUG: format_uri(): /mybucket/?location
DEBUG: Sending request method_string='GET', uri=u'/mybucket/?location', headers={'x-amz-content-sha256': u'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': u'AWS4-HMAC-SHA256 Credential=d6d2206a5638cca2b3d181fe156f8921/20191014/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=6ff26f7f6ac5bc0a8aa245e464f3929d019ceb65edb80d7cf8ddc6084fe9bdd7', 'x-amz-date': '20191014T175314Z'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://backup67:80#1)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Resource>/mybucket/</Resource><RequestId>D1194B1FEDC86D37</RequestId><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 39 31 30 31 34 54 31 37 35 33 31 34 5a 0a 32 30 31 39 31 30 31 34 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 30 30 63 62 62 36 65 61 64 31 30 31 65 63 62 36 39 63 32 64 37 36 61 66 37 63 35 34 65 35 37 31 30 65 38 37 35 34 35 38 65 63 31 63 38 32 62 61 62 61 66 34 32 62 35 35 35 30 33 33 33 38 34 65</StringToSignBytes><SignatureProvided>d6d2206a5638cca2b3d181fe156f8921</SignatureProvided><StringToSign>AWS4-HMAC-SHA256\n20191014T175314Z\n20191014/us-east-1/s3/aws4_request\n00cbb6ead101ecb69c2d76af7c54e5710e875458ec1c82babaf42b555033384e</StringToSign><AWSAccessKeyId>Xadmin@</AWSAccessKeyId></Error>',
 'headers': {'content-length': '1005',
             'content-type': 'application/xml;charset=utf-8',
             'date': 'Mon, 14 Oct 2019 17:53:15 GMT',
             'gateway-protocol': 's3',
             'server': 'CAStor Cluster/11.0.a',
             'via': '1.1 backup67',
             'x-amz-request-id': 'D1194B1FEDC86D37'},
 'reason': 'Forbidden',
 'status': 403}
...

[fviard]

@jamshid : In fact the issue was not to set a port, but when it was set exactly to ":80" or ":443" (depending of your ssl setting).
Anyway, I pushed in MASTER a fix to handle that situation and so everything should now work without encountering the SignatureDoesNotMatch error anymore.
(You can have a look at #1059 if you want more details)

Don't hesitate to re-open / tell me if you still have issues with the last fixes.

[chenxushuo]

fixed？