Unable to fetch bundles: S3 returns 501 Not Implemented

[Hiieu]

Hi there 👋,
I wonder if anyone has a similar issue after upgrading OPA (running on ECS) to v0.47.0:
the application no longer can fetch bundles from S3.
The S3 service sends back an HTTP 501 code: 501 Not Implemented

Short description

After upgrading OPA from v0.46.0 to v0.47.0 and without changing any configurations, the application cannot fetch bundles from the S3 bucket.

We haven't noticed any changes in the GET requests to S3, but for some reason, this error is only returned in v0.47.X. After downgrading the version to 0.46.X OPA is able to fetch bundles without any issues.

Version: v0.47.0
Where OPA is deployed: AWS ECS

OPA config file:

services:
  s3_discovery_bundle:
    url: https://<REDACTED>.s3.eu-west-1.amazonaws.com
    credentials:
      s3_signing:
        metadata_credentials:
          aws_region: eu-west-1
  s3_bundles:
    url: https://<REDACTED>.s3.eu-west-1.amazonaws.com
    credentials:
      s3_signing:
        metadata_credentials:
          aws_region: eu-west-1
discovery:
  service: s3_discovery_bundle
  resource: discovery.tar.gz
  decision: discovery/config
  polling:
    long_polling_timeout_seconds: 45
  signing:
    keyid: discovery_key
keys:
  discovery_key:
    key: <This value is populated by "opa run --set-file">

Command:

["run","--server","--log-level=debug","--log-format=json","--disable-telemetry","--config-file=/tmp/config.yaml","--set-file=keys.discovery_key.key=/tmp/discovery_key"]

Steps To Reproduce

1. Deploy OPA v0.47.X on ECS and use S3 signing.
2. The S3 service should return HTTP 501.

Expected behavior

OPA should be able to fetch bundles from S3 buckets without any issues.

Additional context

Logs from Cloudwatch:

Timestamp | Message | Container
-- | -- | --
06/12/2022, 11:45:31 | {
    "headers": {
        "Content-Type": [
            "application/xml"
        ],
        "Date": [
            "Tue, 06 Dec 2022 11:45:31 GMT"
        ],
        "Server": [
            "AmazonS3"
        ],
        "X-Amz-Id-2": [
            "DsXSPn9+KByKKLUyvwIp9uToGYX2z3z29EzMuCOHr8bVLyvp5I1xCzMwzTM3Ya6zwIs1Hkc5Fc8="
        ],
        "X-Amz-Request-Id": [
            "DXTYQZHBH33R6DQJ"
        ]
    },
    "level": "debug",
    "method": "GET",
    "msg": "Received response.",
    "status": "501 Not Implemented",
    "time": "2022-12-06T11:45:31Z",
    "url": "https://<REDACTED>.s3.eu-west-1.amazonaws.com/discovery.tar.gz"
}
 | opa
06/12/2022, 11:45:31 | {
    "level": "error",
    "msg": "Discovery download failed: server replied with Not Implemented",
    "plugin": "discovery",
    "time": "2022-12-06T11:45:31Z"
}
 | opa
06/12/2022, 11:45:31 | {
    "level": "debug",
    "msg": "Waiting 2m4.462710453s before next download/retry.",
    "time": "2022-12-06T11:45:31Z"
}
 | opa
06/12/2022, 11:45:31 | {
    "level": "debug",
    "msg": "Download starting.",
    "time": "2022-12-06T11:45:31Z"
}
 | opa
06/12/2022, 11:45:31 | {
    "level": "debug",
    "msg": "Signing request with AWS credentials.",
    "time": "2022-12-06T11:45:31Z"
}
 | opa
06/12/2022, 11:45:31 | {
    "level": "debug",
    "msg": "Credentials previously obtained from metadata service still valid.",
    "time": "2022-12-06T11:45:31Z"
}
 | opa
06/12/2022, 11:45:31 | {
    "level": "debug",
    "msg": "awsSigningAuthPlugin:*rest.awsMetadataCredentialService successful",
    "time": "2022-12-06T11:45:31Z"
}
 | opa
06/12/2022, 11:45:31 | {
    "headers": {
        "Authorization": [
            "REDACTED"
        ],
        "Host": [
            "<REDACTED>.s3.eu-west-1.amazonaws.com"
        ],
        "Prefer": [
            "modes=snapshot,delta;wait=45"
        ],
        "User-Agent": [
            "Open Policy Agent/0.47.0 (linux, amd64)"
        ],
        "X-Amz-Content-Sha256": [
            "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
        ],
        "X-Amz-Date": [
            "20221206T114531Z"
        ],
        "X-Amz-Security-Token": [
            "<REDACTED>"
        ]
    },
    "level": "debug",
    "method": "GET",
    "msg": "Sending request.",
    "time": "2022-12-06T11:45:31Z",
    "url": "https://<REDACTED>.s3.eu-west-1.amazonaws.com/discovery.tar.gz"
}

[anderseknert]

Hi @Hiieu 👋 And thanks for reporting this! I wonder if it could be related to the work that went into AWS v4 signing for the http.send built-in function. While they're obviously different things, I think some work was made to have them share code between them. Either way, we'll look into it.

[srenatus]

A header that you provided implies functionality that is not implemented.

https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList

So maybe comparing headers with before/after is a way to start here