Azure - Authentication error when `AZURE_CUSTOM_DOMAIN` set to Azure CDN #1116

JeffreyCA · 2022-02-21T08:24:50Z

Ever since the Azure backend was updated to the new azure-storage-blob library in v1.12, using django-storages with AZURE_CUSTOM_DOMAIN set results in Authentication errors when uploading files. For me this only happens with Akamai CDNs, but as reported by others below it affects other CDN types as well.

v1.12 changed how AZURE_CUSTOM_DOMAIN is used with BlobServiceClient. In v1.11 and earlier, the custom domain was only used to get blob URLs. All other operations like uploading, streaming, getting metadata were being done by making requests to the actual storage account endpoint (https://<accountname>.blob.core.windows.net) even if a custom domain was specified.

In v1.12, the behaviour changed so that the custom domain endpoint is used for all storage operations. This uncovered several different upstream issues causing various storage requests to fail with auth errors.

Issues

1. Auth error due to MAC signature mismatch when `AZURE_CUSTOM_DOMAIN` set to Akamai CDN

Upstream issue: Azure/azure-sdk-for-python#26381

Uploads fail with the following error:

ClientAuthenticationError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
...
authenticationerrordetail:The MAC signature found in the HTTP request '5dC3N7RcRW9V...' is not the same as any computed signature. Server used following string to sign: 'PUT


1

application/octet-stream






x-ms-blob-type:BlockBlob
x-ms-client-request-id:xxxx
x-ms-date:Mon, 21 Feb 2022 07:34:35 GMT
x-ms-version:2020-10-02
Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed 
correctly including the signature.
RequestId:xxx
Time:2022-02-21T07:34:35.1058206Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request '5dC3N7RcRW9V...' is not the same as any computed 
signature. Server used following string to sign: 'PUT

2. Forbidden ClientAuthenticationError when `AZURE_CUSTOM_DOMAIN` set to Microsoft CDN

Upstream issue: Azure/azure-sdk-for-python#23640

Uploads fail with the following error (different than above):

azure.core.exceptions.ClientAuthenticationError: Operation returned an invalid status 'Forbidden'
ErrorCode:AuthenticationFailed

The text was updated successfully, but these errors were encountered:

JeffreyCA · 2022-02-21T22:08:02Z

Opened an issue on Azure Python SDK repo here: Azure/azure-sdk-for-python#23163

madalinpopa · 2022-02-22T09:42:33Z

I have the same issue. Once I update the AZURE_CUSTOM_DOMAIN with my *.azureedge.net domain, I get the error below when run python manage collectstatic.

Traceback (most recent call last):
  File "/home/support/repos/pythonazurestocdn/mysite/manage.py", line 22, in <module>
    main()
  File "/home/support/repos/pythonazurestocdn/mysite/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 425, in execute_from_command_line
    utility.execute()
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 373, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 187, in handle
    collected = self.collect()
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
    handler(path, prefixed_path, storage)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 334, in copy_file
    if not self.delete_file(path, prefixed_path, source_storage):
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 248, in delete_file
    if self.storage.exists(prefixed_path):
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/storages/backends/azure_storage.py", line 241, in exists
    blob_client.get_blob_properties()
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/azure/storage/blob/_blob_client.py", line 1242, in get_blob_properties
    process_storage_error(error)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/azure/storage/blob/_shared/response_handlers.py", line 177, in process_storage_error
    exec("raise error from None")   # pylint: disable=exec-used # nosec
  File "<string>", line 1, in <module>
azure.core.exceptions.ClientAuthenticationError: Operation returned an invalid status 'Forbidden'
ErrorCode:AuthenticationFailed

jschneier · 2022-03-05T04:21:40Z

The upstream issue response seems to indicate this is no longer a problem?

justinpitcher · 2022-05-10T20:09:22Z

It's still happening. We are experiencing the same issue with CDNs and django-storages 1.12.3

dimbleby · 2022-05-14T20:48:57Z

if there's a standalone repro that doesn't involve django-storages, then this is the wrong place to track the problem. Suggest taking it back upstream.

tarak · 2022-06-23T15:00:30Z

The BlobServiceClient uses the custom domain URL, if AZURE_CUSTOM_DOMAIN is set. This obviously won't work for uploading files. See:

django-storages/storages/backends/azure_storage.py

Line 153 in 5015c24

account_domain = self.custom_domain or "{}.blob.{}".format(

IMHO, saving files should use the storage account URL (ending in *.blob.core.windows.net), while the custom domain should only be used if set and return the URL from that custom domain.

Simply put, one cannot write to an Azure CDN endpoint. I may be wrong though...

tarak · 2022-06-23T15:26:17Z

I think that the documentation is misleading. I use an Azure CDN that uses a storage account as endpoint. After I have read the documentation of the Azure backend I thought I could set the AZURE_CUSTOM_DOMAIN setting to the hostname of the CDN endpoint (e.g. edgeaccount.azureedge.net). But now I think that the AZURE_CUSTOM_DOMAIN actually means the custom domain of the storage account itself. If so, please ignore my previous comment...

jeffreykirchner · 2022-07-01T20:23:55Z

I agree with @tarak, docs say AZURE_CUSTOM_DOMAIN can be mycdn.azureedge.net.

It would also make sense for the AZURE_CONNECTION_STRING setting to not override the AZURE_CUSTOM_DOMAIN. Files should be uploaded with the AZURE_CONNECTION_STRING settings and return the AZURE_CUSTOM_DOMAIN url.

Another option would be to add an AZURE_CDN setting that accomplishes the above. Most people are using Azure CDN and Azure Storage in concert.

milopersic · 2022-09-16T20:03:18Z

Having the same issue as described here, setting AZURE_CUSTOM_DOMAIN to the CDN endpoint breaks uploads for my application, but setting it to the storage container itself serves files from the container not the CDN.

JeffreyCA · 2022-09-22T17:52:45Z

Is everyone who has this issue using Akamai CDNs? Regarding the auth error I was having, I discovered the root issue to be with how Akamai CDN handles the If-None-Match HTTP header, which the new Azure SDKs are sending: Azure/azure-sdk-for-python#26381. Not sure if this will be fixed on the SDK side or CDN side.

While I think Azure SDKs are supposed to work with custom domains and CDNs (see Azure/azure-sdk-for-python#25536), there are multiple issues affecting different Azure CDN types which make using CDN endpoints quite unreliable at the moment. django-storages v1.11 (old Azure SDK) uses the storage account URL for API calls, bypassing the custom domain. I think it makes sense to restore that behaviour and have the custom domain only be used for getting the blob URL. This is how the old Azure and S3 backends work.

Issues identified when connecting using CDN endpoint instead of storage URL

Microsoft CDN

Streaming large files leads to IncompleteReadError: Azure Blob Storage: "TypeError: '<' not supported between instances of 'int' and 'NoneType'" #1097
- Issue closed but provided workaround does not work.
ClientAuthenticationError when calling exists(): BlobClient returns ClientAuthenticationError when Azure CDN is enabled Azure/azure-sdk-for-python#23640
- Issue closed with workaround being "use a different CDN type".

Akamai CDN

AuthenticationFailed when uploading: Uploading blob using Akamai CDN with overwrite=False fails with AuthenticationFailed Azure/azure-sdk-for-python#26381

jeffreykirchner · 2022-09-22T18:44:49Z

I am using Microsoft CDN + Azure Storage Account.
I agree, with your proposed fix, see my comment above.

dimbleby · 2022-09-22T18:52:55Z

Did you report the problem that opens this thread to the Azure SDK repository? Since that repro didn't involve django-storages at all, it's clear that any fix would need to be upstream.

Of course it might make sense to implement some workaround while waiting for upstream to sort this. But this is not the only codebase that uses blob storage - it's better for the world if this can be fixed at root!

jeffreykirchner · 2022-09-22T19:22:21Z

The issue is that when STATICFILES_STORAGE is defined, it is using AZURE_CUSTOM_DOMAIN to connect.
DEFAULT_FILE_STORAGE is working correctly and does not use AZURE_CUSTOM_DOMAIN to connect.
Perhaps it would make sense to add a new AZURE_CDN_URL parameter that would handle CDN endpoints. That would leave AZURE_CUSTOM_DOMAIN to define an actual custom domain (and not an endpoint URL).

JeffreyCA · 2022-09-22T23:12:21Z

I created a PR #1176 to revert back to the old behaviour so that AZURE_CUSTOM_DOMAIN is used only for generating file URLs. This is in line with the other storage backends. I also updated the docs to clarify this

mlindemu · 2022-10-17T18:29:57Z

@jschneier , when will we get a new PyPi release with this change rolled in? Thank you for the great library!

mlindemu · 2023-02-20T14:08:06Z

For anyone following or looking at this thread, the fix is in the published 1.13.2 version. Thank you!

JeffreyCA mentioned this issue Sep 22, 2022

[azure] Use AZURE_CUSTOM_DOMAIN only for retrieving blob URLs, and use storage URL for other operations #1176

Merged

jschneier closed this as completed in #1176 Sep 27, 2022

vanbuiten mentioned this issue May 19, 2023

Bumped django-storages from 1.12.3 to 1.13.2 Amsterdam/signals#1240

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure - Authentication error when `AZURE_CUSTOM_DOMAIN` set to Azure CDN #1116

Azure - Authentication error when `AZURE_CUSTOM_DOMAIN` set to Azure CDN #1116

JeffreyCA commented Feb 21, 2022 •

edited

JeffreyCA commented Feb 21, 2022

madalinpopa commented Feb 22, 2022

jschneier commented Mar 5, 2022

justinpitcher commented May 10, 2022

dimbleby commented May 14, 2022

tarak commented Jun 23, 2022

tarak commented Jun 23, 2022 •

edited

jeffreykirchner commented Jul 1, 2022

milopersic commented Sep 16, 2022

JeffreyCA commented Sep 22, 2022 •

edited

jeffreykirchner commented Sep 22, 2022

dimbleby commented Sep 22, 2022 •

edited

jeffreykirchner commented Sep 22, 2022 •

edited

JeffreyCA commented Sep 22, 2022

mlindemu commented Oct 17, 2022 •

edited

mlindemu commented Feb 20, 2023

Azure - Authentication error when AZURE_CUSTOM_DOMAIN set to Azure CDN #1116

Azure - Authentication error when AZURE_CUSTOM_DOMAIN set to Azure CDN #1116

Comments

JeffreyCA commented Feb 21, 2022 • edited

Issues

1. Auth error due to MAC signature mismatch when AZURE_CUSTOM_DOMAIN set to Akamai CDN

2. Forbidden ClientAuthenticationError when AZURE_CUSTOM_DOMAIN set to Microsoft CDN

JeffreyCA commented Feb 21, 2022

madalinpopa commented Feb 22, 2022

jschneier commented Mar 5, 2022

justinpitcher commented May 10, 2022

dimbleby commented May 14, 2022

tarak commented Jun 23, 2022

tarak commented Jun 23, 2022 • edited

jeffreykirchner commented Jul 1, 2022

milopersic commented Sep 16, 2022

JeffreyCA commented Sep 22, 2022 • edited

Issues identified when connecting using CDN endpoint instead of storage URL

Microsoft CDN

Akamai CDN

jeffreykirchner commented Sep 22, 2022

dimbleby commented Sep 22, 2022 • edited

jeffreykirchner commented Sep 22, 2022 • edited

JeffreyCA commented Sep 22, 2022

mlindemu commented Oct 17, 2022 • edited

mlindemu commented Feb 20, 2023

Azure - Authentication error when `AZURE_CUSTOM_DOMAIN` set to Azure CDN #1116

Azure - Authentication error when `AZURE_CUSTOM_DOMAIN` set to Azure CDN #1116

JeffreyCA commented Feb 21, 2022 •

edited

1. Auth error due to MAC signature mismatch when `AZURE_CUSTOM_DOMAIN` set to Akamai CDN

2. Forbidden ClientAuthenticationError when `AZURE_CUSTOM_DOMAIN` set to Microsoft CDN

tarak commented Jun 23, 2022 •

edited

JeffreyCA commented Sep 22, 2022 •

edited

dimbleby commented Sep 22, 2022 •

edited

jeffreykirchner commented Sep 22, 2022 •

edited

mlindemu commented Oct 17, 2022 •

edited