Bad encoding in query parameters with new TestClient using httpx.Client

[azogue]

After the change to httpx for the TestClient in v0.21.0 (), the query parameters are not properly encoded? when sending requests with it, and strings are corrupted when received in the endpoints.

Quick example to show the problem:

import logging
from fastapi import FastAPI

app = FastAPI()

@app.get("/example")
async def _show_encoding_error(look_for: str):
    return {"found": look_for}

if __name__ == '__main__':
    from starlette.testclient import TestClient

    with TestClient(app) as client:
        params = {"look_for": "plain text"}
        resp = client.get("/example", params=params).json()
        logging.warning(resp)
        assert resp["found"] == "plain text"

        params = {"look_for": "España"}
        resp = client.get("/example", params=params).json()
        logging.warning(resp)
        assert resp["found"] == "España", resp["found"]

The example app works as expected if called from the SwaggerUI or from another python process using a plain httpx.Client, so it appears something broke with the new wrapping for TestClient 🥲

import httpx

params = {"look_for": "España"}
with httpx.Client(base_url="http://localhost:8000/") as client:
    resp = client.get("/example", params=params).json()
    assert resp["found"] == "España"

Details

First issued in fastapi repo: tiangolo/fastapi#5646, before checking the TestClient comes from here 🙈

Environment:

• macOS M1 arm64
• fastapi v0.87.0
• starlette v0.21.0
• httpx v0.23.0

[Kludex]

cc @tomchristie

[robotadam]

We found a similar issue with the new TestClient when passing a date time with timezone parameter. I wrote up a test that executes this:

def test_query_param(test_client_factory):
    def homepage(request):
        return Response(request.query_params["dt"])

    app = Starlette(routes=[Route("/", endpoint=homepage)])
    dt = "2020-07-14T00:00:00+00:00"
    client = test_client_factory(app)
    response = client.get("/", params={"dt": dt})
    assert response.text == dt

E       AssertionError: assert '2020-07-14T00:00:00 00:00' == '2020-07-14T00:00:00+00:00'
E         - 2020-07-14T00:00:00+00:00
E         ?                    ^
E         + 2020-07-14T00:00:00 00:00
E         ?

The cause appears to be https://github.com/encode/starlette/blob/master/starlette/testclient.py#L202, where the query string is decoded from ascii and unquoted. Unquoting before a query string parse appears to remove the +; removing that line lets it through correctly.

Removing the unquote looks like it does the trick for España as well.

[azogue]

where the query string is decoded from ascii and unquoted

This is it 👍 @robotadam
I also saw this yesterday, and checked that, in _TestClientTransport.handle_request,

Removing the unquote looks like it does the trick

    def handle_request(self, request: httpx.Request) -> httpx.Response:
        scheme = request.url.scheme
        netloc = unquote(request.url.netloc.decode(encoding="ascii"))
        path = request.url.path
        raw_path = request.url.raw_path
        # query = unquote(request.url.query.decode(encoding="ascii"))
        query = request.url.query.decode(encoding="ascii")

but not sure why is it there?

@Kludex , should we upgrade this discussion to a new issue?

[Kludex]

It looks like GitHub doesn't notify people when the issue is created from a discussion... #1952

[azogue]

Oh, sorry, it was my fault, too many emails in inbox 😅

[azogue]

I just checked that removing both unquote operations for the netloc and the query parts does work as expected and does not break any of the tests 🥳 (609 passed, 2 xfailed in 2.41s in quick venv with py3.10)

So it could be as simple as that to fix this 🤩