How to disable URL path normalization?

[heroesm]

Background

In vulnerability scanning scenario we often need to use dot segments /../ in URL path to test server vulnerability.

Test case

URL

http://httpbin.org/anything/2/../1

Comparison

With curl we can:

curl --path-as-is 'http://httpbin.org/anything/2/../1'

With requests we can:

psf/requests#5289

import requests

url = "http://httpbin.org/anything/2/../1"
s = requests.Session()
req = requests.Request(method='GET' ,url=url)
prep = req.prepare()
prep.url = url
r = s.send(prep, verify=False)
print(r.json()['url'])
# print `http://httpbin.org/anything/2/../1`

With aiohttp we can:

https://stackoverflow.com/questions/61140190/python-force-aiohttp-to-not-normalize-url

async def arun():
    url = yarl.URL("http://httpbin.org/anything/2/../1", encoded=True)
    async with aiohttp.ClientSession() as session:
        async with session.get(url, allow_redirects=False) as resp:
            print((await resp.json())['url'])
# print `http://httpbin.org/anything/2/../1`

How should we achieve the same effect with httpx?

[x0day]

as note: aio-libs/aiohttp#4307

[heroesm]

Thank you for your reply.

If we did want to provide support for this at a higher level, we might want to lean on raw_path

It would also be very helpful.

The high level API of httpx is convenient and I would like to take advantage of these while make it possilbe to turn off URL normalization.

Right now I have been used another hack:

import httpx


class NonNormalisedURL(httpx.URL):
    @property
    def is_absolute_url(self):
        return False


url = NonNormalisedURL('http://httpbin.org/anything/2/../1')

print(httpx.get(url).json()['url'])

Then use a switch option in the wrapper code to decide how to replace original url with NonNormalisedURL.

Would it have some serious side effects? Would it be broken in future version?

[tomchristie]

Would it have some serious side effects? Would it be broken in future version?

You'd need to take responsibility for those aspects yourself. Probably you'd be fine.
Pinning your requirements strictly, and reviewing upgrades wouldn't be a bad idea tho'.

[heroesm]

Thank you.

If someday related high level API change like raw_path described above is about to take place, please leave some comments here. I would be very grateful.

[jcdevil]

Hello,
Re-opening this discussion as I currently have the exact same need.
But, given this discussion took place almost 2 years ago : was there any change related to that requirement ?
Thanks for your help and answers.

[heroesm]

Hello, Re-opening this discussion as I currently have the exact same need. But, given this discussion took place almost 2 years ago : was there any change related to that requirement ? Thanks for your help and answers.

I'm still using the workaround mentioned above and limiting the HTTPX version to httpx>=0.18,<0.19. I don't know if it applies to the latest version or if there are any other recommended approaches now.

Additionally, I've also patched the rfc3986 library：

import rfc3986
rfc3986.misc.NON_PCT_ENCODED.update(['|', '{', '}', '[', ']'])
# patch query pattern in rfc3986 to allow `[]{}` in query
# see also https://github.com/encode/httpx/issues/1833
from rfc3986 import abnf_regexp
abnf_regexp.QUERY_RE = (
    '^([/?:@' + r'\[\]\{\}' + abnf_regexp.UNRESERVED_RE + abnf_regexp.SUB_DELIMITERS_RE + ']|%s)*$' % abnf_regexp.PCT_ENCODED
)

rfc3986.misc.RELATIVE_REF_MATCHER = re.compile(r'^%s(\?%s)?(#%s)?$' % (
    abnf_regexp.RELATIVE_PART_RE,
    abnf_regexp.QUERY_RE,
    abnf_regexp.FRAGMENT_RE,
))

rfc3986.misc.ABSOLUTE_URI_MATCHER = re.compile(r'^%s://' % (
    abnf_regexp.COMPONENT_PATTERN_DICT['scheme'],
))

[jakkdl]

Giving this another +1, was about to create a new issue/discussion but found this one just before posting the below.

what

Add an option to httpx.URL similar to yarl.URL encoded=True

>>> import httpx
>>> httpx.URL('http://0.0.0.0.0/././b')
URL('http://0.0.0.0.0/b')

>>> import yarl
>>> yarl.URL('http://0.0.0.0.0/././b')
URL('http://0.0.0.0.0/b')
>>> yarl.URL('http://0.0.0.0.0/././b', encoded=True)
URL('http://0.0.0.0.0/././b')

why

I'm working on transitioning aiobotocore to httpx from aiohttp and hit a roadblock due to the above issue since:

1. AWS S3 allows periods and slashes in object key names, and the key name is specified by sending it as part of the URL. So these periods and slashes cannot be removed without changing behavior.
2. I don't have control over percent-escaping, which happens in botocore/, where they explicitly mark / as safe. I could try to tackle the issue on that front instead, but thought I'd try it here first.

I'm open to writing a PR that implements it, it feels like it should be a fairly straightforward implementation given that the normalization already is conditional:

httpx/httpx/_urlparse.py

Lines 256 to 257 in 392dbe4

	if has_authority:
	path = normalize_path(path)

doing hacky things with unsupported APIs and strictly pinning requirements would not be great, given that this is a library and not an application.

[tomchristie]

The target extension provides this functionality...

https://www.encode.io/httpcore/extensions/#target

Looks like the httpx docs need updating to also include it.

[jakkdl]

Thank you! Yeah even reading that section of the docs now I wouldn't guess that's what I or the OP needed in this situation. Though at least now if you search "url normalize" in httpx discussion you'll hit this comment 🎉

[tomchristie]

Perhaps someone would be up for submitting a PR adding the target extension to the docs.

I wouldn't guess that's what I or the OP needed in this situation

That's useful yep, we could probably draw from the existing httpcore docs here, and in addition highlight example use-cases.

[tomchristie]

Docs pull request... #3160