Add streaming multipart/form-data encoder

[alexwlchan]

This is an attempt to get the streaming multipart/form-data encoder landed again (for #51, #624). This is something that was previously worked on in #2331 and has stalled; I'd like to have a go at getting it across the line.

Status: still fairly experimental. I wanted to open a PR to get it running in the proper CI environment, but I'm still wrapping my head around the code – a bunch of the comments are me talking to myself saying "hmm, look at this further". But I think I'm gradually moving forward! 🥰

---

So far:

• I've merged the latest main into sigmavirus24’s branch. This was mostly clean; the only issues were:

  • The project now uses import typing; typing.X rather than from typing import X; X, so I had to change all of those (but reading the comments on the original PR, I think that was the preference anyway?)
  • The definition of _TYPE_BODY has moved from connection.py to _base_connection.py

  Otherwise I haven't touched the original code.

• Run nox -rs lint, nox -rs mypy and nox -rs docs to fix various style/linting issues. Again I don't think these are especially meaningful.

• I've added some new tests to increase the coverage. I think I'm up to 100% for the decoder, but not the encoder.

• I read the review comments on the original PR. I couldn't see anything outstanding, but if I've missed something I'm happy to incorporate those changes into this new branch.

• This code comes from requests-toolbelt. I had a quick look at the code in the requests-toolbelt repo to see if there are any changes between the original PR and now that we should incorporate into this urllib3 copy, but there aren't any.

Still to do:

• Increase the coverage to 100%. I don't think I'm quite there yet, but I want to get it running in CI and hopefully find what still needs doing
• Any amendments from code review in 2024

Because the original code already had several rounds of review, I'll add some inline comments to highlight my new changes, to make it easier for reviewers.

[alexwlchan]

This test is new, to add coverage for this error.

[alexwlchan]

This test is new, to add coverage for the content_length property.

[alexwlchan]

This test is new, to add coverage for a couple of lines in the multipart encoder:

if isinstance(file_type, bytes):
    file_type = file_type.decode("utf-8")

[alexwlchan]

I'm getting a CI failure for Python 3.13 on Windows, but I don't think it's related to this patch?

Failed to build cffi
ERROR: Could not build wheels for cffi, which is required to install pyproject.toml-based projects

I can see a similar failure in the latest build on main.

[alexwlchan]

This line isn't currently tested, and I need to think more about what the intention is here, because I haven't found a way to trigger this branch.

In particular neither an open file nor an io.BytesIO return True here, so I wonder if a different check is what's meant (if this branch is even needed at all?):

import io
import typing

with open(__file__, 'rb') as f:
    print(isinstance(f, typing.BinaryIO))  # False

bytes_io = io.BytesIO(b'Hello, World!')
print(isinstance(bytes_io, typing.BinaryIO))  # False

[sigmavirus24]

Maybe bytearray? I can't remember honestly

[alexwlchan]

Another note to investigate: if I run this locally, I just get {"field": "value"} as the output; filename.txt is uploaded as a file rather than a form-encoded field. 🤔

[sigmavirus24]

❤️ Thank you @alexwlchan ! I don't have the time with the kiddo and hopefully you can claim the bounty (since I couldn't even if I got the PR over the line)

[alexwlchan]

(Having picked this up I've immediately stalled due to some stuff in my personal life, but I do want to continue this after this week is over!

[sigmavirus24]

By the way, requests/toolbelt#316 is a reasonable addition to the enclosed but will complicate this a smidge

[alexwlchan]

I've had a more thorough read of decoder.py and left some comments, which I'll push a commit to address shortly.

[alexwlchan]

I wonder what happens if you receive a value which isn't properly quoted – e.g. if value starts with an open quote but there's no matching close quote.

[alexwlchan]

This and the test case in test_encoder.py are the only tests in the project that subclass unittest.TestCase.

I'm going to leave them as-is for now, but I wonder if they should be rewritten to use pytest fixtures or similar, to match the rest of the project.

[alexwlchan]

I've made a couple more tweaks, and I think the decoder is now working. In particular I created a small Flask server that uses requests-toolbelt to send multipart-encoded responses, so I can test the code in my urllib3 fork:

from flask import Flask, Response
from requests_toolbelt import MultipartEncoder

app = Flask(__name__)

@app.route('/downloads')
def downloads():
    m = MultipartEncoder(
           fields={'field0': 'value', 'field1': 'value',
                   'field2': ('filename', open('file.py', 'rb'), 'text/plain')}
        )

    s = m.to_string()

    return Response(s, mimetype=m.content_type, headers={"content-type": m.content_type})

app.run(debug=True)

Next up is diving into the encoder code and seeing what else needs to be done there.