Tilde characters in URL paths should be treated as an unreserved character. (~)

[ImBatou]

Checklist

• The bug is reproducible against the latest release and/or master.
• There are no similar issues or pull requests to fix it yet.

Describe the bug

Footlocker for example doesn't accept the percent encode of the tilde, it means that you cannot request the page

To reproduce

Burp on this page for example, you will get a 301 everytime, because they don't accept the percent encode of the tilde, you can take a look at it if you use a MITM like Charles or Burp
https://www.footlocker.pt/product/~/315345197602.html

Expected behavior

request on this URL : https://www.footlocker.pt/product/~/315345197602.html
(MITM on chrome)

Actual behavior

request on this URL : https://www.footlocker.pt/product/%2F/315345197602.html
(MITM on httpx)

Debugging material

Any MITM software

Environment

• OS: Windows
• Python version: 3.8.5
• HTTPX version: 0.17.1
• Async environment: N/A
• HTTP proxy: no
• Custom certificates: no

[ImBatou]

Ok I did some research, and the rfc3986 module has been updated for this problem on github but not in pip, everything is working while loading the new rfc3986 manually.

[tomchristie]

@LeBatou Great thanks!

Referenced issue is python-hyper/rfc3986#74

We might want to check in with the rfc3986 team and see what we can do about:

• Helping get the test suite passing again on that repo.
• Any help we can provide in order to get a release rolled.

(Clearly there's also some longer term thinking to do about how we can try to build on our sponsor model to help downstream projects too.)

[tomchristie]

And... resolved! ☺️

See rfc3986 1.15 - https://pypi.org/project/rfc3986/#history

Before upgrade...

>>> httpx.get('https://www.footlocker.pt/product/~/315345197602.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_api.py", line 201, in get
    trust_env=trust_env,
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_api.py", line 110, in request
    allow_redirects=allow_redirects,
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_client.py", line 748, in request
    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_client.py", line 836, in send
    history=[],
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_client.py", line 870, in _send_handling_auth
    history=history,
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_client.py", line 897, in _send_handling_redirects
    "Exceeded maximum allowed redirects.", request=request
httpx.TooManyRedirects: Exceeded maximum allowed redirects.

Then either update rfc3986 directly...

$ pip install -U rfc3986

Or update httpx, and request dependencies to update where possible too....

$ pip install -U httpx --upgrade-strategy eager

Which then gives us this...

>>> import httpx
>>> httpx.get('https://www.footlocker.pt/product/~/315345197602.html')
<Response [200 OK]>